Tag Archives: GitHub

Searching code, repos, issues, pull requests and users on GitHub using PowerShell

Coding, especially in PowerShell, is not about remembering the exact syntax of how to do something, it is more about knowing how to try things out and how to find the information you need to do whatever you need to accomplish.

Quite often, even though we don’t remember the exact syntax, we know the thing we want to do is something we’ve already done before. So there is a good chance it’s already in the code base we have published on GitHub over the years.
If only we could use GitHub as an extension of our brain and, even better, directly from our PowerShell console …

This is why I wrote the PowerShell module PSGithubSearch, available on the PowerShell Gallery and (off course) on GitHub. It uses the extensive GitHub Search API to search the following items on GitHub.com :

  • Code
  • Repositories
  • Issues
  • Pull requests
  • Users
  • Organisations

Let’s have a look at a few examples to illustrate how we can use its 4 cmdlets.

Find-GitHubCode :

Maybe you are working on a new PowerShell function which could really use a -Confirm parameter, let’s say Invoke-NuclearRedButton.
You know that this requires the SupportsShouldProcess cmdletBinding attribute but you never ever remember exactly how to use it inside the function. Then, you could do that :

C:\> Find-GitHubCode -Keywords 'SupportsShouldProcess' -User $Me -Language 'PowerShell' | Select-Object -First 1


FileName               : Update-ChocolateyPackage.psm1
FilePath               : Update-ChocolateyPackage/Update-ChocolateyPackage.psm1
File Url               : https://github.com/MathieuBuisson/Powershell-Utility/blob/5dde8e9bb4fa6244953fee4042e2100acfc6
                         ad72/Update-ChocolateyPackage/Update-ChocolateyPackage.psm1
File Details Url       : https://api.github.com/repositories/28262426/contents/Update-ChocolateyPackage/Update-Chocolat
                         eyPackage.psm1?ref=5dde8e9bb4fa6244953fee4042e2100acfc6ad72
Repository Name        : MathieuBuisson/Powershell-Utility
Repository Description : Various generic tools (scripts or modules) which can be reused from other scripts or modules
Repository Url         : https://github.com/MathieuBuisson/Powershell-Utility

 

OK, but you might want to see the actual file content with the code. This is not built into the cmdlet because it requires an additional API call and the GitHub search API limits the number of unauthenticated requests to 10 per minute, so I tried to limit as much as possible the number of API requests. Anyway, here is how to get the actual code :

C:\> $FileUrl = (Find-GitHubCode -Keywords 'SupportsShouldProcess' -User $Me -Language 'PowerShell' | Select-Object -First 1).url
C:\> $Base64FileContent = (Invoke-RestMethod -Uri $FileUrl).Content
C:\> [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($Base64FileContent)) -split '\n' |
 Where-Object { $_ -match 'ShouldProcess' }

    [CmdletBinding(SupportsShouldProcess)]
                If ($PSCmdlet.ShouldProcess($($CurrentPackage.Name), "Install-Package")) {

 
Yep, not that easy.
The original result provides us with a URL that we can query to get more information on that file.
This additional query gives us the content of the file but as a Base64 encoded string, that’s why we need to decode it as UTF8.
But then, it is still a single (large) string, that’s why we needed to split it on each new line character (\n).

This is not very practical for large files, so you can restrict the search to files of a certain size (in bytes), like so :

C:\> Find-GitHubCode -Repo "$Me/Powershell-Utility" -Keywords 'SupportsShouldProcess' -SizeBytes '<4000'


FileName               : Update-ChocolateyPackage.psm1
FilePath               : Update-ChocolateyPackage/Update-ChocolateyPackage.psm1
File Url               : https://github.com/MathieuBuisson/Powershell-Utility/blob/5dde8e9bb4fa6244953fee4042e2100acfc6
                         ad72/Update-ChocolateyPackage/Update-ChocolateyPackage.psm1
File Details Url       : https://api.github.com/repositories/28262426/contents/Update-ChocolateyPackage/Update-Chocolat
                         eyPackage.psm1?ref=5dde8e9bb4fa6244953fee4042e2100acfc6ad72
Repository Name        : MathieuBuisson/Powershell-Utility
Repository Description : Various generic tools (scripts or modules) which can be reused from other scripts or modules
Repository Url         : https://github.com/MathieuBuisson/Powershell-Utility

 

There are other parameters which can help narrow down the search to get exactly what we need. We’ll leave that as homework, because we have 3 other cmdlets to cover.

Find-GitHubRepository :

As an example, we might want to know what interesting project one of our favorite automation hero is currently working on :

C:\> $WarrenRepos = Find-GitHubRepository -Keywords 'PowerShell' -User 'RamblingCookieMonster'
C:\> $WarrenRepos | Sort-Object -Property pushed_at -Descending | Select-Object -First 1


Name         : PSDepend
Full Name    : RamblingCookieMonster/PSDepend
Owner        : RamblingCookieMonster
Url          : https://github.com/RamblingCookieMonster/PSDepend
Description  : PowerShell Dependency Handler
Fork         : False
Last Updated : 2016-08-28T15:48:52Z
Last Pushed  : 2016-09-29T16:06:04Z
Clone Url    : https://github.com/RamblingCookieMonster/PSDepend.git
Size (KB)    : 202
Stars        : 10
Language     : PowerShell
Forks        : 2

 
Warren has been working on this new project for a good month, and this project already has 10 stars and 2 forks. It definitely looks like an interesting project.

We can also sort the results by popularity (number of stars) or by activity (number of forks). For example, we could get the 5 most popular PowerShell projects related to deployment, like so :

C:\> $DeploymentProjects = Find-GitHubRepository -Keywords 'Deployment' -In description -Language 'PowerShell' -SortBy stars
C:\> $DeploymentProjects | Select-Object -First 5

Name         : AzureStack-QuickStart-Templates
Full Name    : Azure/AzureStack-QuickStart-Templates
Owner        : Azure
Url          : https://github.com/Azure/AzureStack-QuickStart-Templates
Description  : Quick start ARM templates that deploy on Microsoft Azure Stack
Fork         : False
Last Updated : 2016-10-01T19:16:20Z
Last Pushed  : 2016-09-28T22:03:44Z
Clone Url    : https://github.com/Azure/AzureStack-QuickStart-Templates.git
Size (KB)    : 11617
Stars        : 118
Language     : PowerShell
Forks        : 74

Name         : unfold
Full Name    : thomasvm/unfold
Owner        : thomasvm
Url          : https://github.com/thomasvm/unfold
Description  : Powershell-based deployment solution for .net web applications
Fork         : False
Last Updated : 2016-09-25T03:55:03Z
Last Pushed  : 2014-10-10T07:28:22Z
Clone Url    : https://github.com/thomasvm/unfold.git
Size (KB)    : 1023
Stars        : 107
Language     : PowerShell
Forks        : 13

Name         : AppRolla
Full Name    : appveyor/AppRolla
Owner        : appveyor
Url          : https://github.com/appveyor/AppRolla
Description  : PowerShell framework for deploying distributed .NET applications to multi-server environments inspired
               by Capistrano
Fork         : False
Last Updated : 2016-09-20T10:29:29Z
Last Pushed  : 2013-08-25T18:04:13Z
Clone Url    : https://github.com/appveyor/AppRolla.git
Size (KB)    : 546
Stars        : 91
Language     : PowerShell
Forks        : 10

Name         : SharePointDsc
Full Name    : PowerShell/SharePointDsc
Owner        : PowerShell
Url          : https://github.com/PowerShell/SharePointDsc
Description  : The SharePointDsc PowerShell module provides DSC resources that can be used to deploy and manage a
               SharePoint farm
Fork         : False
Last Updated : 2016-09-05T12:05:18Z
Last Pushed  : 2016-09-27T17:07:10Z
Clone Url    : https://github.com/PowerShell/SharePointDsc.git
Size (KB)    : 3576
Stars        : 80
Language     : PowerShell
Forks        : 48

Name         : PSDeploy
Full Name    : RamblingCookieMonster/PSDeploy
Owner        : RamblingCookieMonster
Url          : https://github.com/RamblingCookieMonster/PSDeploy
Description  : Simple PowerShell based deployments
Fork         : False
Last Updated : 2016-09-27T17:33:51Z
Last Pushed  : 2016-09-29T09:42:46Z
Clone Url    : https://github.com/RamblingCookieMonster/PSDeploy.git
Size (KB)    : 665
Stars        : 76
Language     : PowerShell
Forks        : 18

 
We used the In parameter to look for the “Deployment” keyword in the description field of repositories, we could look only in the repositories name or ReadMe, if we wanted to.

Now let’s dive into the heart of GitHub collaboration with issues and pull requests.

Find-GitHubIssue :

Maybe, we want to know the most commented open issue on the PowerShell project which hasn’t been assigned to anyone yet. This is pretty easy to do :

C:\> Find-GitHubIssue -Repo 'PowerShell/PowerShell' -Type issue -State open -No assignee -SortBy comments |
 Select-Object -First 1


Title        : Parameter binding problem with ValueFromRemainingArguments in PS functions
Number       : 2035
Id           : 172737066
Url          : https://github.com/PowerShell/PowerShell/issues/2035
Opened by    : dlwyatt
Labels       : {Area-Language, Issue-Bug, Issue-Discussion}
State        : open
Assignees    :
Comments     :
Created      : 2016-08-23T15:51:55Z
Last Updated : 2016-09-29T14:45:43Z
Closed       :
Body         : Steps to reproduce
               ------------------

               Define a PowerShell function with an array parameter using the ValueFromRemainingArguments property of
               the Parameter attribute.  Instead of sending multiple arguments, send that parameter a single array
               argument.

               ```posh
                & {
                    param(
                        [string]
                        [Parameter(Position=0)]
                        $Root,

                        [string[]]
                        [Parameter(Position=1, ValueFromRemainingArguments)]
                        $Extra)
                    $Extra.Count;
                    for ($i = 0; $i -lt $Extra.Count; $i++)
                    {
                       "${i}: $($Extra[$i])"
                    }
                } root aa,bb
               ```

               Expected behavior
               -----------------
               The array should be bound to the parameter just as you sent it, the same way it works for cmdlets.
               (The "ValueFromRemainingArguments" behavior isn't used, in this case, it should just bind like any
               other array parameter type.)  The output of the above script block should be:

               2
               0: aa
               1: bb

               Actual behavior
               ---------------
               PowerShell appears to be performing type conversion on the argument to treat the array as a single
               element of the parameter's array, instead of checking first to see if more arguments will be bound as
               "remaining arguments" first.  The output of the above script block is currently:

               1
               0: aa bb

               Additional information
               ------------------

               To demonstrate that the behavior of cmdlets is different, you can use this code:

               ```posh
               Add-Type -OutputAssembly $env:temp\testBinding.dll -TypeDefinition @'
                   using System;
                   using System.Management.Automation;

                   [Cmdlet("Test", "Binding")]
                   public class TestBindingCommand : PSCmdlet
                   {
                       [Parameter(Position = 0)]
                       public string Root { get; set; }

                       [Parameter(Position = 1, ValueFromRemainingArguments = true)]
                       public string[] Extra { get; set; }

                       protected override void ProcessRecord()
                       {
                           WriteObject(Extra.Length);
                           for (int i = 0; i < Extra.Length; i++)
                           {
                               WriteObject(String.Format("{0}: {1}", i, Extra[i]));
                           }
                       }
                   }
               '@

               Import-Module $env:temp\testBinding.dll

               Test-Binding root aa,bb
               ```

               Environment data
               ----------------

               <!-- provide the output of $PSVersionTable -->

               ```powershell
               > $PSVersionTable
               Name                           Value
               ----                           -----
               PSEdition                      Core
               PSVersion                      6.0.0-alpha
               PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
               WSManStackVersion              3.0
               GitCommitId                    v6.0.0-alpha.9-107-g203ace04c09dbbc1ac00d6b497849cb69cc919fb-dirty
               PSRemotingProtocolVersion      2.3
               CLRVersion
               SerializationVersion           1.1.0.1
               BuildVersion                   3.0.0.0
               ```

 
That is a pretty detailed and complex issue.

Now, we might wonder who is the top contributor (in terms of Pull requests) to the PowerShell documentation project :

C:\> $DocsPR = Find-GitHubIssue -Type pr -Repo 'PowerShell/PowerShell-Docs'
C:\> $DocsPR | Group-Object { $_.user.login } |
 Sort-Object -Property Count -Descending | Select-Object -First 10

Count Name                      Group
----- ----                      -----
  141 eslesar                   {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/650; repositor...
   81 HemantMahawar             {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/656; repositor...
   59 alexandair                {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/627; repositor...
   32 juanpablojofre            {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/657; repositor...
   28 neemas                    {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/107; repositor...
   26 joeyaiello                {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/554; repositor...
   20 Dan1el42                  {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/645; repositor...
   10 PlagueHO                  {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/330; repositor...
    9 JKeithB                   {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/605; repositor...
    8 saldana                   {@{url=https://api.github.com/repos/PowerShell/PowerShell-Docs/issues/455; repositor...
   

 
Wow, some people are really into documentation.

Find-GithubUser :

GitHub.com is the largest open-source software community, so it is a great place to find passionate and talented coders who are willing to share their craft with the community.

Let’s say you are a recruiter or a hiring manager and you are looking for a PowerShell talent in Ireland. The cmdlet Find-GithubUser can facilitate that search :

C:\> Find-GithubUser -Type user -Language 'PowerShell' -Location 'Ireland' | Where-Object { $_.Hireable }


UserName      : JunSmith
Full Name     : Jun Smith
Type          : User
Url           : https://github.com/JunSmith
Company       :
Blog          :
Location      : Ireland
Email Address :
Hireable      : True
Bio           :
Repos         : 12
Gists         : 0
Followers     : 7
Following     : 4
Joined        : 2015-01-17T13:27:24Z

UserName      : TheMasterPrawn
Full Name     : Rob Allen
Type          : User
Url           : https://github.com/TheMasterPrawn
Company       : Unity Technology Solutions IRL
Blog          :
Location      : Ireland
Email Address :
Hireable      : True
Bio           : I.T/Dev guy, gamer, geek living and working in Ireland.
Repos         : 3
Gists         : 0
Followers     : 0
Following     : 0
Joined        : 2014-06-10T11:09:20Z

 
2 users only. That’s helpful … It highlights the huge PowerShell skill gap we have in Ireland.

Let’s focus on UK and organizations, then. Organizations don’t have followers (or following) so we cannot filter them on the number of followers they have, but we can narrow down the search to those which have 5 or more repos :

C:\> Find-GithubUser -Type org -Language 'PowerShell' -Location 'UK' -Repos '>=5'


UserName      : SpottedHyenaUK
Full Name     : Spotted Hyena UK
Type          : Organization
Url           : https://github.com/SpottedHyenaUK
Company       :
Blog          : http://www.spottedhyena.co.uk
Location      : Leeds, UK
Email Address :
Hireable      :
Bio           :
Repos         : 5
Gists         : 0
Followers     : 0
Following     : 0
Joined        : 2015-02-03T14:38:16Z

UserName      : VirtualEngine
Full Name     : Virtual Engine
Type          : Organization
Url           : https://github.com/VirtualEngine
Company       :
Blog          : http://virtualengine.co.uk
Location      : UK
Email Address : info@virtualengine.co.uk
Hireable      :
Bio           :
Repos         : 17
Gists         : 0
Followers     : 0
Following     : 0
Joined        : 2014-03-21T14:51:14Z

 
That was just a few examples, but since each of these cmdlets have many parameters, this was just scratching the surface of what can be done with them.
For the full list of parameters (and how to use them), please refer to the README file on GitHub or use Get-Help.

Also, I’m aware it’s not perfect (yet) so issues and pull requests are very much welcome.

Generate a GitHub README.md file from your PowerShell module help

When publishing a script or a module to a GitHub repository, it is a best practice to add a README.md file in the same folder as a script file. That way, anyone who stumbles upon the script on GitHub is presented with a brief documentation of what the script does and how to use it.

I write exclusively PowerShell modules (.psm1) nowadays, as opposed to scripts (.ps1). The main reasons for that are : I want to build tools which behaves just like native cmdlets and I want my tools to be easily reusable. Documentation is an important part of making a module reusable by anyone.

Even though I’m a big believer in documentation, sometimes, I have a hard time practicing what I preach. After working for a few hours on a module, I just want to upload it and be done with it. Exploring, tinkering and scripting in the shell and/or the ISE is the fun part, writing the documentation is the boring part. But wait, what do we do with the boring stuff ?
We AU… TO… MATE it !

If you lookup on the internet, you will find PowerShell scripts which do this, but they are using the content of the file and parsing it to extract the desired information. Text-parsing in PowerShell… Ugh. It’s so against nature, it makes me feel dirty.

Usually, I already have comment-based help in the module, so let’s use it by importing the module and leveraging the object-oriented output of Get-Help. As a bonus, this approach will work with any PowerShell module (script or compiled) and any help (comment-based or XML-based).

PS C:\> Import-Module "C:\GitHub\Powershell-Utility\Example\Example.psm1"
PS C:\> $FullModulePath = Resolve-Path -Path "C:\GitHub\Powershell-Utility\Example\Example.psm1"
PS C:\> $Module = Get-Module | Where-Object { $_.Path -eq $FullModulePath }
PS C:\> $Module

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     0.0        Example                             {Get-Nothing, Set-Nothing}

 
To illustrate how to do this, we are working with a module called Example.

The README.md is a text file so, from a PowerShell perspective, it will be an array of strings. To prepare it, it’s as simple as that :

PS C:\> $Readme = @()

 
It is empty, but we are going to populate it as we go.

PS C:\> $Commands = Get-Command -Module $Module
PS C:\> $Readme += "##Description :"
PS C:\> $Readme += "`n`r"
PS C:\> $CommandsCount = $($Commands.Count)
PS C:\> $Readme += "This module contains $CommandsCount cmdlets :  "
PS C:\> $Readme
##Description :

This module contains 2 cmdlets :

 
First, we got the command(s) in the module and store them in $Commands.
“##” is a markdown tag to mark the following text as a second level heading, like h2 in HTML.
`n is the new line character and `r is the carriage return.
By the way, `n doesn’t work well in GitHub markdown. So here is a trick : to start a new line without starting a new paragraph, you need to add 2 trailing spaces at the end of the line.

Then, $Commands.Count gives us the number of commands in the module. We store it in the variable $CommandsCount, which allows us to document the number of cmdlets in the module.

Then, we add the name of each cmdlet :

Foreach ($Command in $Commands) {
    $Readme += "**$($Command.Name)**  "
} 

 
"**" is a markdown tag. The text surrounded by "**" will be in bold.

Now, I want to document the PowerShell version required by the module. This is not in the help, but this is specified in the #Requires statement at the very beginning of the module, so we can get to it this way :

$FirstLine = $Module.Definition -split "`n" | Select-Object -First 1
If ($FirstLine -like "#Requires*") {
    $PSVersionRequired = $($FirstLine -split " " | Select-Object -Last 1)
}
If ($PSVersionRequired) {
    $Readme += "It requires PowerShell version $PSVersionRequired (or later)."
}

 
Now, let’s start the cmdlet-specific section :

PS C:\> $Readme += "`n`r"
PS C:\> $Name = $Commands[0].Name
PS C:\> $Readme += "##$Name :"
PS C:\> $Readme += "`n`r"
PS C:\> $HelpInfo = Get-Help $Name -Full
PS C:\> $Readme += $HelpInfo.description
PS C:\> $Readme
##Description :

This module contains 2 cmdlets :
It requires PowerShell version 4 (or later).

##Get-Nothing :

This cmdlet does absolutely nothing and does it remarkably well.
It takes objects as input and it outputs nothing.

 
We run Get-Help for the first cmdlet in our module, using the -Full parameter to get everything from the Help and store that in a variable $HelpInfo.
Then, the description of the cmdlet is easily retrieved from the description property of $HelpInfo.

Then, let’s get to the parameters section of our documentation.

For each parameter, we want to document its name, description, and if it has a default value, a user-friendly sentence explaining its default value. It turns out that everything we need is in the “parameters” property of our variable $HelpInfo :

PS C:\> $CommandParams = $HelpInfo.parameters.parameter
PS C:\> $CommandParams

-InputObject <PSObject[]>
    Specifies one or more object(s) to get.
    It can be string(s), integer(s), file(s), any type of object.

    Required?                    true
    Position?                    1
    Default value                (Get-Item *)
    Accept pipeline input?       true (ByValue)
    Accept wildcard characters?  false


-Filter <String>
    Specifies a filter in the provider's format or language. The value of this parameter qualifies
    the InputObject.
    The syntax of the filter, including the use of wildcards, or regular expressions, depends on
    the provider.

    Required?                    false
    Position?                    2
    Default value
    Accept pipeline input?       false
    Accept wildcard characters?  false

 

Let’s add this to the README :

PS C:\> $Readme += "###Parameters :"
PS C:\> $Readme += "`n`r"
PS C:\> Foreach ($CommandParam in $CommandParams) {
           $Readme += "**" + $($CommandParam.Name) + " :** " + $($CommandParam.description.Text) + "  "

           If ( $($CommandParam.defaultValue) ) {
               $ParamDefault = $($CommandParam.defaultValue).ToString()
               $Readme += "If not specified, it defaults to $ParamDefault ."
           }
           $Readme += "`n`r"
}
PS C:\>

 
The default value for a parameter might not be a string, so we used the method ToString() which can convert any object to a string.

I usually give 2 or 3 usage examples in the comment-based help and this is the case in our Example module. So let’s add these examples into our README :

PS C:\> $Readme += "###Examples :`n`r"
PS C:\> $Readme += $HelpInfo.examples | Out-String

 
And here is how it looks like on GitHub :

README Screenshot

You can see it for yourselves here.

With these little tools in my belt, I wrote a PowerShell Module named ReadmeFromHelp and it is available on GitHub. It contains one cmdlet New-ReadmeFromHelp and the README.md for this module has been automatically generated by… itself.