How to use multithreading in PowerShell with custom Modules

in this blog post, I will go through how you can use Start-Threadjob to use multithreading in your PowerShell script. I will show how you can pass your own custom modules into the PowerShell thread jobs created.

The reason why I started looking into this functionality is that I have to build a custom module for interacting with the Cisco Meraki APIv1. I am then creating a script that will query all the networks in the Meraki Organisation and then look for all networks with a specific VLAN.

If I were to build this script regularly with no multithreading it would do every single API call one by one. If you have let’s say 50 networks in your organization this could take a decent amount of time to run through all the networks and run the different API calls. Now with multithreading, we can generate multiple API calls simultaneously and increase the speed of running through all the networks.

Now take in mind that I am using some custom PowerShell cmdlet from my custom PowerShell module. If you are interested in the Module you can find the repository on my Github account. Even though I’m showcasing this functionality with my specific problem, you should be able to use the PowerShell functionality and use it with your own problem.

Start-ThreadJob

For using multithreading in my scripts I usually go with Start-ThreadJob. Some of the key things you need to be aware of when working with multithreading are:

Everything you do inside the Thread Job is running in a separate workspace:

  • This means that if you are connected to an API, Azure, Office365, or some other service you will probably need to reconnect inside the Thread Job.
  • It also means that if you have a function or module loaded in your session, you will need to reload it inside the Thread Job

Data generated inside the Thread Job needs to be exported into your current session:

  • When you run a Job in a separate thread and the job generates an output with some data, you will need to export the data out of the job to be able to use it in your main session.

You might need to throttle the number of threads generated:

  • If you have 50 Meraki networks for example and you create 50 threads where each thread will call the API you will receive the status code 429 meaning that you have exceeded the number of concurrent sessions available. This is fairly common with API’s that you are limited to only 5 concurrent sessions. To prevent this you can throttle how many threads you want to have open at the same time.

A quick show case of Start-ThreadJob

I will start by just creating a quick demonstration of how to can use Start-ThreadJob.

In this example, I have a function that will wait 1 second for 30 seconds. For every second it will output the count of the second reached. If I run the command and measure it I see that it takes just about 30 seconds to run the command:

function TestJobs {
    $array = @(1..30)

    foreach($i in $array){
        #Write-Output -InputObject $i
        Start-Sleep -s 1
    }
}


Measure-Command {TestJobs}

Output:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 30
Milliseconds      : 77
Ticks             : 300777990
TotalDays         : 0.000348122673611111
TotalHours        : 0.00835494416666667
TotalMinutes      : 0.50129665
TotalSeconds      : 30.077799
TotalMilliseconds : 30077.799

Now to speed up this command I could use the Start-ThreadJob command like shown below:

function TestJobs {
    $array = @(1..30)

    foreach($x in $array) {
        Start-ThreadJob -InputObject $x -ScriptBlock {
            Start-Sleep -s 1
        } -ThrottleLimit 30
    }

    $runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
    While($runningJobs -ne 0){
        $runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
    }

    Get-Job | Remove-Job
}

Measure-Command {TestJobs}

This function is similar to the previous function in that it has an array of numbers counting from 1 to 30. but instead of running the Start-Sleep function, wait for 1 second and then continue to the next integer in the array and do the same again, it will start 30 jobs.

Each job will run the function Start-Sleep and wait for 1 second, but since all jobs are executed at once the foreach loop has run, the time is minimized to approximately 1 second. This also shows from the output of running the function inside Measure-Command

Output:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 890
Ticks             : 18904049
TotalDays         : 2.18796863425926E-05
TotalHours        : 0.000525112472222222
TotalMinutes      : 0.0315067483333333
TotalSeconds      : 1.8904049
TotalMilliseconds : 1890.4049

Now bear in mind that if you adjusted the -ThrottleLimit parameter, on the Start-ThreadJob function, down the time would increase because it would use fewer threads to complete the 30 jobs.

Explaining the function

Now for people completely new to Start-ThreadJob, I will just quickly explain the code in the function.

The function starts by creating an Array of numbers increasing from 1 to 30:

$array = @(1..30)

It then runs a foreach loop, wherein each loop will start a thread job. The current number in the loop, from the array, defined by $x will be passed to the thread job, by the parameter -InputObject.

The actual code which is run inside the thread job is defined by the -ScriptBlock parameter.
To use the $x variable I passed to the thread job, I can call it by using the variable $input, inside the thread job.

I tell the function Start-ThreadJob to use 30 simultaneous threads by specifying the parameter -ThrottleLimit 30.

foreach($x in $array) {
	Start-ThreadJob -InputObject $x -ScriptBlock {
		Start-Sleep -s 1
	} -ThrottleLimit 30
}

After the foreach loop, I use the command Get-Job and search for all jobs which have a state of either “Running” or “NotStarted”. I then use the command inside a while loop. and if the count of jobs that are either running or haven’t been started yet gets to 0, I then clean up and removes all the jobs, since they are finished. If you don’t clean up all the jobs you would just have a lot of jobs in your session with the state as “Completed”

$runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
While($runningJobs -ne 0){
	$runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
}

Get-Job | Remove-Job

Retreiving data from the ThreadJobs

You can retreive the data you produce in each job by running the command:

Receive-Job

Now in the function, inside each thread job, the number from the array will be outputted. This could also be some data you retrieved from an API, the process is the same. To retrieve this data from the thread job and use it in my current PowerShell session I can inside the foreach loop, retrieve the data and store it inside another Array, and then output that array from the function. To Showcase this I have added the line:

Write-Output "Thread DATA: $($input)"

Which will be created inside each thread job

I have then created a new foreach loop which will run through all the completed jobs, retrieve the data, and add the data to the new array I have created before the second foreach loop is created.

$threadData = New-Object -TypeName System.Collections.ArrayList

Get-Job | % {$data = Receive-Job; $threadData.Add($data)}

I then finish the function by adding:

return $threadData

The entire function will look like below. Bear in mind that I have added the code $null = Start-ThreadJob -InputObject …. The reason for this is that I don’t want the command start-threadjob to output anything to the console. I only want the Write-Output statement to output.

function TestJobs {
    $array = @(1..30)

    foreach($x in $array) {
        $null = Start-ThreadJob -InputObject $x -ScriptBlock {
            Start-Sleep -s 1
            Write-Output -InputObject "Thread DATA: $($input)"
        } -ThrottleLimit 30
    }

    $runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
    While($runningJobs -ne 0){
        $runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
    }

    $threadData = New-Object -TypeName System.Collections.ArrayList
    Get-Job | Foreach-Object {$data = Receive-Job -Id $_.Id; $null = $threadData.Add($data)}

    Get-Job | Remove-Job

    return $threadData
}

Measure-Command {$data = TestJobs}

$data

Now when I run the function it will an output similar to below:

Thread DATA: 1
Thread DATA: 2
Thread DATA: 3
Thread DATA: 4
Thread DATA: 5
Thread DATA: 6
	...

Passing a function or a module to your Start-ThreadJob

When you are running the function Start-ThreadJob you will create a new run space that can be compared to a completely new PowerShell session. So if you are using a custom module in your PowerShell script, and you need to use some of the functions from this module inside the thread jobs, you will need to import the module or function in the thread job.

To do this you can call Start-ThreadJob with the parameter -InitializationScript. This parameter will let you run a script to initialize the new run space. You can use this script to import the module or function you need or to connect to an API or service.

In the example, I will show I have created a module with a simple function. The function takes an integer as input and multiplies that integer with 1000. It then returns an object containing the old number and the new number.

function ThreadModuleFunction {
    [CmdletBinding()]
    param (
        [Parameter()]
        [Int]$Number
    )

    $num = $Number * 1000
    $object = New-Object -TypeName psobject -Property @{
        OldNumber = $Number
        NewNumber = $num
    }

    return $object
}

Export-ModuleMember -function ThreadModuleFunction

Calling the function looks like this:

ThreadModuleFunction -Number 33

Output:

OldNumber NewNumber           
--------- ---------                                        
       33     33000 

Starting a threadjob and importing the module into the thread runspace

Now for using the function inside my three jobs, I will use the parameter -InitializationScript, and I will use the parameter -ArgumentList instead of -InputObject.

When I use -ArgumentList I can call the object inside the thread job by using $args[0]. If I had multiple objects to pass into the thread job I would call it like below:

Start-ThreadJob -ArgumentList $object1 $object 2 -ScriptBlock {
    Write-Output -InputObject "$($args[0]) -  Calling Object1"
    Write-Output -InputObject "$($args[1]) -  Calling Object2"
}

Now the script I can use for calling my custom functions inside my custom module, inside my thread job would look like below:

$array = @(1..30)
foreach($x in $array) {
    $null = Start-ThreadJob -ArgumentList $x -InitializationScript {Import-Module "./threadModule.psm1"} -ScriptBlock {
        Start-Sleep -s 1
        $runData = ThreadModuleFunction -Number $args[0]
        Write-Output -InputObject $runData
    } -ThrottleLimit 30
}

$runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
While($runningJobs -ne 0){
    $runningJobs = (Get-Job | ? {($_.State -eq "Running") -or ($_.State -eq "NotStarted")}).count
}

$threadData = New-Object -TypeName System.Collections.ArrayList
Get-Job | Foreach-Object {$data = Receive-Job -Id $_.Id; $null = $threadData.Add($data)}

Get-job | Remove-job

This code above is just similar to the function I showed earlier, except this time I initialize each thread job with the script:

Import-Module "./threadModule.psm1"

Now when I run the script shown above it will give the output from each the module function run in each thread and looking like the output below:

OldNumber NewNumber
--------- ---------
        1      1000
        2      2000
        3      3000
        4      4000
        5      5000
        6      6000
        7      7000
    ...

You could also use dot sourcing if you want to initialize another script into the run space by using:

. ./scriptfolder/script1.ps1

Conclusion

So to sum up this blog post. You can import your custom modules and functions really easily by using the parameter -InitializationScript when running Start-ThreadJob.

You can add objects from your current PowerShell session into the thread job by using -InputObject or -ArgumentList.

When using InputObject you can only pass one object to the function and you can use the object inside the thread job with the variable $input

When using -ArgumentList you can pass multiple objects to your thread job adding the objects after the parameter like shown below:

Start-ThreadJob -ArgumentList $object1 $object2

And you can use the objects inside the thread job by calling them with $args[0]. The ArgumentList is working like an Array where the first object will be [0] the second object will be [1] and so on.

And at last, if you need to be connected to either an API or a service for running the functionality inside your thread job, you will need to connect again inside the thread job either by the -InitializationScript or by running the connection commands inside the new thread.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

This website uses cookies. By continuing to use this site, you accept our use of cookies.