• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

Multithreaded Powershell Remote Event Log Collection Example

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

I.M.O.G.

Glorious Leader
Joined
Nov 12, 2002
Location
Rootstown, OH
Building off my previous thread giving powershell event log examples, I had mentioned I needed to multithread it but I hadn't gotten around to actually doing it. I took a few minutes this morning, and it was really simple to do that and this is how.

This is the powershell script that pulls the events I want from remote machines, and it accepts a computer name as an argument. I saved it as Get-Event7001Failed.ps1

Code:
$computer = $args[0]

$days = (Get-Date) - (New-TimeSpan -Day 90)

If(Test-Connection -ComputerName $computer -Quiet){
Get-WinEvent -FilterHashtable @{logname='Microsoft-Windows-GroupPolicy/Operational'; id=7001; StartTime=$days} -EA SilentlyContinue -cn $computer | where-object  { $_.Message -like '*failed*' } | select @{label='TimeCreated';expression={$_.TimeCreated.ToString("yyyy-M-d HH:mm:ss")}},MachineName,@{n='Message';e={$_.Message -replace '\s+', " "}}}

This is the powershell script that handles the multithreading (credit: http://www.get-blog.com/?p=189). I saved this script as Run-CommandMultiThreaded.ps1:

Code:
#.Synopsis
#    This is a quick and open-ended script multi-threader searcher
#    
#.Description
#    This script will allow any general, external script to be multithreaded by providing a single
#    argument to that script and opening it in a seperate thread.  It works as a filter in the 
#    pipeline, or as a standalone script.  It will read the argument either from the pipeline
#    or from a filename provided.  It will send the results of the child script down the pipeline,
#    so it is best to use a script that returns some sort of object.
#
#    Authored by Ryan Witschger - http://www.Get-Blog.com
#    
#.PARAMETER Command
#    This is where you provide the PowerShell Cmdlet / Script file that you want to multithread.  
#    You can also choose a built in cmdlet.  Keep in mind that your script.  This script is read into 
#    a scriptblock, so any unforeseen errors are likely caused by the conversion to a script block.
#    
#.PARAMETER ObjectList
#    The objectlist represents the arguments that are provided to the child script.  This is an open ended
#    argument and can take a single object from the pipeline, an array, a collection, or a file name.  The 
#    multithreading script does it's best to find out which you have provided and handle it as such.  
#    If you would like to provide a file, then the file is read with one object on each line and will 
#    be provided as is to the script you are running as a string.  If this is not desired, then use an array.
#    
#.PARAMETER InputParam
#    This allows you to specify the parameter for which your input objects are to be evaluated.  As an example, 
#    if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to 
#    find all processes where the name was the provided computername and fail.  You need to specify that the 
#    parameter that you are providing is the "ComputerName".
#
#.PARAMETER AddParam
#    This allows you to specify additional parameters to the running command.  For instance, if you are trying
#    to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
#    parameter.  This command takes a hash pair formatted as follows:  
#
#    @{"ParameterName" = "Value"}
#    @{"ParameterName" = "Value" ; "ParameterTwo" = "Value2"}
#
#.PARAMETER AddSwitch
#    This allows you to add additional switches to the command you are running.  For instance, you may want 
#    to include "RequiredServices" to the "Get-Service" cmdlet.  This parameter will take a single string, or 
#    an aray of strings as follows:
#
#    "RequiredServices"
#    @("RequiredServices", "DependentServices")
#
#.PARAMETER MaxThreads
#    This is the maximum number of threads to run at any given time.  If resources are too congested try lowering
#    this number.  The default value is 20.
#    
#.PARAMETER SleepTimer
#    This is the time between cycles of the child process detection cycle.  The default value is 200ms.  If CPU 
#    utilization is high then you can consider increasing this delay.  If the child script takes a long time to
#    run, then you might increase this value to around 1000 (or 1 second in the detection cycle).
#
#    
#.EXAMPLE
#    Both of these will execute the script named ServerInfo.ps1 and provide each of the server names in AllServers.txt
#    while providing the results to the screen.  The results will be the output of the child script.
#    
#    gc AllServers.txt | .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1
#    .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1 -ObjectList (gc .\AllServers.txt)
#
#.EXAMPLE
#    The following demonstrates the use of the AddParam statement
#    
#    $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -InputParam ComputerName -AddParam @{"Name" = "BITS"}
#    
#.EXAMPLE
#    The following demonstrates the use of the AddSwitch statement
#    
#    $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -AddSwitch @("RequiredServices", "DependentServices")
#
#.EXAMPLE
#    The following demonstrates the use of the script in the pipeline
#    
#    $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -InputParam ComputerName -AddParam @{"Name" = "BITS"} | Select Status, MachineName
#


Param($Command = $(Read-Host "Enter the script file"), 
    [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList,
    $InputParam = $Null,
    $MaxThreads = 20,
    $SleepTimer = 200,
    $MaxResultTime = 120,
    [HashTable]$AddParam = @{},
    [Array]$AddSwitch = @()
)

Begin{
    $ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
    $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
    $RunspacePool.Open()
        
    If ($(Get-Command | Select-Object Name) -match $Command){
        $Code = $Null
    }Else{
        $OFS = "`r`n"
        $Code = [ScriptBlock]::Create($(Get-Content $Command))
        Remove-Variable OFS
    }
    $Jobs = @()
}

Process{
    Write-Progress -Activity "Preloading threads" -Status "Starting Job $($jobs.count)"
    ForEach ($Object in $ObjectList){
        If ($Code -eq $Null){
            $PowershellThread = [powershell]::Create().AddCommand($Command)
        }Else{
            $PowershellThread = [powershell]::Create().AddScript($Code)
        }
        If ($InputParam -ne $Null){
            $PowershellThread.AddParameter($InputParam, $Object.ToString()) | out-null
        }Else{
            $PowershellThread.AddArgument($Object.ToString()) | out-null
        }
        ForEach($Key in $AddParam.Keys){
            $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
        }
        ForEach($Switch in $AddSwitch){
            $Switch
            $PowershellThread.AddParameter($Switch) | out-null
        }
        $PowershellThread.RunspacePool = $RunspacePool
        $Handle = $PowershellThread.BeginInvoke()
        $Job = "" | Select-Object Handle, Thread, object
        $Job.Handle = $Handle
        $Job.Thread = $PowershellThread
        $Job.Object = $Object.ToString()
        $Jobs += $Job
    }
        
}

End{
    $ResultTimer = Get-Date
    While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0)  {
    
        $Remaining = "$($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).object)"
        If ($Remaining.Length -gt 60){
            $Remaining = $Remaining.Substring(0,60) + "..."
        }
        Write-Progress `
            -Activity "Waiting for Jobs - $($MaxThreads - $($RunspacePool.GetAvailableRunspaces())) of $MaxThreads threads running" `
            -PercentComplete (($Jobs.count - $($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).count)) / $Jobs.Count * 100) `
            -Status "$(@($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False})).count) remaining - $remaining" 

        ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
            $Job.Thread.EndInvoke($Job.Handle)
            $Job.Thread.Dispose()
            $Job.Thread = $Null
            $Job.Handle = $Null
            $ResultTimer = Get-Date
        }
        If (($(Get-Date) - $ResultTimer).totalseconds -gt $MaxResultTime){
            Write-Error "Child script appears to be frozen, try increasing MaxResultTime"
            Exit
        }
        Start-Sleep -Milliseconds $SleepTimer
        
    } 
    $RunspacePool.Close() | Out-Null
    $RunspacePool.Dispose() | Out-Null
}

So to fire off my little event log script, I just do the following which runs the multithreading script, feeds in my event log script as a command, the target list of computers as an objectlist, and tells it to run 50 threads at a time:
.\Run-CommandMultiThreaded.ps1 -Command .\Get-Event7001Failed.ps1 -ObjectList (gc C:\targets.txt) -MaxThreads 50


This completes in a few minutes now running against 1300 machines, instead of taking a couple hours. Output is the same as before, except I get about 900 lines back instead of these few lines I'm posting as an example:

Code:
2016-1-18 09:53:15 computername User logon policy processing failed for domain\user in 1 seconds.   
2016-1-18 09:47:04 computername2 User logon policy processing failed for domain\user2 in 1 seconds.   
2016-1-18 09:45:26 computername3 User logon policy processing failed for domain\user3 in 1 seconds.  
2016-1-18 09:43:28 computername4 User logon policy processing failed for domain\user4 in 1 seconds.   
2016-1-18 09:42:04 computername5 User logon policy processing failed for domain\user5 in 1 seconds.
 
I'm wondering if I can leverage the multi-threading feature combined with the GCI Cmdlet to finally replace LS.EXE as the program I used to scan and record usage stats for our command's shared drives.

We have about 25 network drives for various departments and commands on my base. I currently kick off 24 instances of LS.EXE (which records full path+filename, size, modified date, and parent folder information to a text file which I then import into a MS Access database to pull the stats needed for monthly metric reports. (aged data, space used by division/department, Pivot tables by division by filetypes or filesizes as well as pulling stats based on things like personal files, databases, picture files, multimedia files, etc)

I like LS.EXE because it's simple to use but kicking off 24 instances is a bit tedious and it currently takes 2.4 days for the scan to complete on our largest shared drive (and there's no resume feature so if we have a power hiccup I have to start the scan over from scratch)

I used to use VB code to do this when our network drives were geographically local but, because our shared drives were re-homed to Philly from Memphis, the latency has increased to the point that the VB script requires 500 contiguous hours to complete, LOL.
 
I'm certain you could. The specifics to do it right would take a little thought, perhaps a couple revisions, but just doing it wouldn't be hard.

Doing it wrong should be really simple though. You should be able to automate a script to launch your 24 instances of ls.exe really easily. This would just recreate exactly what you are manually doing now. It wouldn't get done any quicker but it wouldn't be cumbersome to kick off anymore.

To do it right, at first thought, I would expect what you'd want is a loop that recursively enumerates all (sub)folders of the shares, and then kicks off children processes to compile the data you need for all the file contents of those folders. The key part I would focus on would be to include logging to indicate how long different parts of the script are taking to execute, so you know where the long running steps are - once you have that info, you just have to decide how to slice that up and make it a parallel task.

So the key there would just be learning the limiting factors, and optimizing around them to make this perform quickly. Whenever possible, executing the task from a more local resource is probably preferred if that's an option in your situation. Like if the server admins could run it local to the server and output results to a directory whee you can download, that would be great. I'm assuming the bureaucracy of your situation prevents that from being a possibility however, or you wouldn't be doing something as crazy as you are currently.
 
On a separate note, we are still chasing around an issue with failed network connectivity in our environment.

Our issue is that machines retain valid TCPIP info from DHCP, but lose all network connectivity. They can't be contacted, and if logged into locally they can't ping their default gateway. This does not affect every machine, but less than 10%. It occurs at every site out of 10 separate facilities. It isn't model specific, or image/install date specific. It is most likely the fault of some infrastructure task breaking connectivity, however this is a black box from my teams perspective (I mamage our help desk, we have limited access beyond PCs). Infrastructure says its a PC problem, but we are demonstrating to the contrary by providing better data to them. A reboot or reconnecting the network cable immediately resolves the connectivity problem (tcpip info does not change, same lease, same IP, etc - only difference is it works).

At the end user side, what happens is when they login in the mornings their login hangs, and they are unable to work until they force a restart. After restart they are good to go.

I've been collecting and reporting on the group policy processing failures thst indicate an enduser experienced this, to indicate a high level of production impact to warrant a weekly department meeting until resolved. Each result returned by the script indicates a prior loss of connectivity that has caused a production downtime at the time the event was logged. Generally 40-80 employees are affected each day. Less than 10%, but considerable and consistent downtime really pisses off department heads. Our user population is also such that they take advantage of this, so what may be a 5 minute warranted dowtime due to technical issues becomes a 30 minute downtime because they shuck their responsibilities, aren't well supervised, and blame it on IT issues.

Currently I'm also beginning reporting on system event ID 1014. It indicates DNS resolution errors, which when compiled from every machine in the company, are building a case that the actual loss of connectivity occurs at consistent intervals on certain days at certain sites, and at the same time across many different affected machines. This is helpful, because the group policy processing failures are a lagging indicator - the loss of connection prior to that happening could have been days prior and nonone attempted to login on the machine until now. The DNS resolution failures however are logged much closer to the time of network connectivity loss, as even when idle machines are regularly trying to chat with active directory and related endpoint management services, so when their connectivity fails DNS resolution erros are logged soon after.

So anyway, something is broken in our environment. I suspect it's Sophos related. I've never seen a PC with fully valid TCPIP info from DHCP with zero connectivity. But you reconnect the network cable, settings stay the same, and everything works perfectly normal again.

So that's one real world use case for stuff like this.

Another use case is ensuring PCs pull policy and prefer contacting their LAN domain controllers... My team doesn't have access to AD sites and services to ensure networks/site links/costs are properly configured (they aren't - which is why my team doesn't have access, as the admin does not appreciate constructive feedback). But thru GP logging, I am able to pull events demonstrating widespread and consistent records of machines traversing WAN links to talk to DCs in different states instead of talking to the DC in the closet 10 feet away. I was blown off several times trying to raise a concern, until I had records to share then it got addressed the same week. All machines now prefer their LAN DC, although I'm still not sure the config follows Best practice, as we still have some delays at restart suggesting machines may be timing out attempting WAN connections still... Battle for another day. ;)
 
Last edited:
Back