Rambling Cookie Monster

Winlogbeat and pipelines and painless scripts, oh my!

2019-05-18T07:00:00+00:00

Overview

Rambling
Overview
Winlogbeat
Ingest pipelines
What now?

Rambling

Whew! It’s been about 2 weeks since the 2019 PowerShell + DevOps Global Summit - the first conference I’ve played a role in organizing. It was super fun, and it was an honor to get a chance to help with content for the summit… that said, it’s been a nice two weeks of recovering!

A lot of folks asked what I do nowadays! I gave my usual :headtilt: computers? response, but the gist of it is, while I still do quite a bit with PowerShell, I have a bunch of other building blocks to design solutions with! One of the more fun ones has been Elastic Stack.

So! Today we’ll dive into using Winlogbeat and ingest pipelines, in case it saves anyone else from wading through disparate documentation with no clear layout of an end-to-end example.

Disclaimer: I know way, way less about logging and Elastic Stack than I do about PowerShell.

Overview

Hopefully, you already have structured logging. If you don’t, or if you’re paying Splunk-level-money, my condolences.

Things we won’t be talking about today:

How important structured logging is, the many critical use cases that it enables, or any of that jazz. I assume you know it’s important
Spinning up a cluster, or even a single test node. You can find that elsewhere
PowerShell

So… what are we going to talk about? We’re going to show a slightly-more-useful config than the vanilla Winlogbeat examples you see out there, including:

how to filter log sources before sending them on to Elasticsearch
how to point logs at ingest pipelines
how to create an ingest pipeline

Let’s get going!

Winlogbeat

We tend to use Elastic Beats clients - there are other options out there, including NXLog.

We’ll use Winlogbeat in this example. There are a few pieces we’ll dive into:

what logs we care about
whether any of the logs need to be processed in a pipeline
whether to process anything on the client

What we care about

So! First, we’ll tell Winlogbeat what we care about, and tack on information about which pipeline to ship logs to.

winlogbeat.event_logs:
  - name: Security
    ignore_older: 168h
    fields:
      rc_ingest_pipeline: 'windows'
      rc_index_name: 'windows'
    fields_under_root: true
  - name: 'Microsoft-Windows-DNS-Client/Operational'
    ignore_older: 168h
    processors:
      - drop_event.when.not:
          and:
          - equals.event_id: 3008
          - and:
            - not.equals.event_data.QueryOptions: '140737488355328'
            - not.equals.event_data.QueryResults: ''
  - name: System
    ignore_older: 168h
    processors:
      - drop_event.when.not:
          or:
          - equals.source_name: 'Microsoft-Windows-Eventlog'
          - and:
            - equals.source_name: 'Microsoft-Windows-GroupPolicy'
            - equals.level: Error
            - or:
              - equals.event_id: 1085
              - equals.event_id: 1125
              - equals.event_id: 1127
              - equals.event_id: 1129
          - and:
            - equals.source_name: 'Microsoft-Windows-Eventlog'
            - equals.level: Information
            - equals.event_id: 104

This looks ugly but it gives you an idea of the flexibility you can get, even if you stick to yaml.

In this example:

I want some operational DNS client log, but only if they are event id 3008, and aren’t associated with empty or local machine name resolution
I want everything from the security log
I want three varieties (or) of events from the System log

You can nest and and or statements, which gives you some nice flexibility. I err on the side of collecting too much, but these filters at least get me started.

Point to a pipeline and index

Now! You might have noticed something odd in the Security bit:

  - name: Security
    ignore_older: 168h
    fields:
      rc_ingest_pipeline: 'windows'
      rc_index_name: 'windows'
    fields_under_root: true

This is saying that for every event in the Security event log, I want to tack on an rc_ingest_pipeline field, with the value windows (fields_under_root just means I don’t want a nested property). This is how I tell Winlogbeat which ingest pipeline I want to use. In this case, I just have a generic catch-all windows pipeline, but you’ll likely have many others.

Jumping ahead, in the output section, you’ll see that we use the field we just created to point to a pipeline:

output.elasticsearch:
  hosts: ["https://REDACTED01:9200", "https://REDACTED02:9200"]
  username: "REDACTED"
  password: "REDACTED"
  ssl.certificate_authorities: ["C:\\beats\\ca.crt"]
  pipeline: "%{[rc_ingest_pipeline]}"
  index: "%{[rc_index_name]}"

By using "%{[rc_ingest_pipeline]}", we’re evaluating to whatever the value of the event’s rc_ingest_pipeline field is - note that we could have defined different values for different log sources. There are several other ways to route events, including pipelines with conditions, or just a hard coding output.elasticsearch.pipeline rather than using the variable expansion we used.

Do some more client side processing

We’re almost done with the config! There’s one last spot we can have some fun: Processors! In this example, I just remove the message field from some of our most common and verbose events (i.e. those that chew up storage without much benefit):

processors:
  - drop_fields:
      when:
        or:
          - equals.event_id: 4624
          - equals.event_id: 4634
          - equals.event_id: 4625
          - equals.event_id: 4648
          - equals.event_id: 4768
          - equals.event_id: 4658
          - equals.event_id: 4776
      fields: ["message"]

That’s about it! Now, lets look at that windows ingest pipeline we’re routing some events to.

Ingest pipelines

Ingest nodes let you process things. They’re generally more efficient, but less observable than Logstash pipelines. Do consider the implications; personally, I would tend to lean towards Logstash in most scenarios. Oh well!

First, we need to define the pipeline. You can use a number of processors, including grok and geopip, but for these events, I just want to run a script, and I want to know if any errors come up (tip: they will, particularly on your first iterations of writing a script)

{
    "description": "Process Windows event log",
    "processors": [
        {
            "script": {
                "id": "windows"
            }
        }
    ],
    "on_failure": [
        {
            "set": {
                "field": "pipeline_error",
                "value": "true"
            }
        },
        {
            "set" : {
                "field" : "pipeline_error_message",
                "value" : "{{ _ingest.on_failure_message }}"
              }
        },
        {
            "set" : {
                "field" : "pipeline_error_type",
                "value" : "{{ _ingest.on_failure_processor_type }}"
              }
        }
    ]
}

All I have is a pointer that says run this script called 'windows'!, and some error handling to give me info on whether an error occurred, the error message, and the error type.

This is important. You want a quick way to see if any events aren’t being processed correctly - by adding these fields, you can search for and alert on any processing errors.

Right. Maybe don’t alert on errors when you fail to grok a field that’s based on user input!

Next, we need to actually define the script. We’ll be using painless. This is a misnomer.

I like a little context. I don’t want the whole message field, I just want a brief what is this event for?. I also don’t memorize enums, so I translate things like logon type.

if( ctx.log_name == 'Security' && ctx.event_data.LogonType != null ) {
    def logonmap =
    [
        '0': 'System',
        '2': 'Interactive',
        '3': 'Network',
        '4': 'Batch',
        '5': 'Services',
        '7': 'Unlock',
        '8': 'NetworkClearText',
        '9': 'NewCredentials',
        '10': 'RemoteInteractive',
        '11': 'Cached'
    ];
    def key = ctx.event_data.LogonType;
    def newvalue = logonmap[key];
    if (newvalue != null)
    {
        ctx.rc_logon_type = newvalue;
    }
}

# Defining this without initializing it as a map broke things
if( ctx.log_name == 'Security' ) {
    Map eventmap = new HashMap();

    # subset of event ids for the sake of brevity....
    eventmap.put('4720', 'A user account was created.');
    eventmap.put('4722', 'A user account was enabled.');
    eventmap.put('4723', 'An attempt was made to change an account password.');
    eventmap.put('4724', 'An attempt was made to reset an account password.');
    eventmap.put('4725', 'A user account was disabled.');
    eventmap.put('4726', 'A user account was deleted.');
    eventmap.put('4727', 'A security-enabled global group was created.');
    eventmap.put('4728', 'A member was added to a security-enabled global group.');
    eventmap.put('4729', 'A member was removed from a security-enabled global group.');
    eventmap.put('4730', 'A security-enabled global group was deleted.');
    eventmap.put('4731', 'A security-enabled local group was created.');

    def key = String.valueOf(ctx.event_id);
    def newvalue = eventmap[key];
    if (newvalue != null)
    {
        ctx.rc_description = newvalue;
    }
}

That’s it! In the script, I make some assumptions and create fields to indicate the description of the event, and to have a friendly name for the logon type. I like context and convenience - you might consider adding much more than this : )

Do note that I’m doing the same thing - assigning a value based on a hashmap - in two ways. I ran into errors when I tried the simplified syntax on a hashmap with ~420 keys, and switching syntax fixed things.

What now?

There’s a lot of flexibility here!

The same Winlogbeat config syntax for pipelines can be used in Filebeat
You might explore parsing the windows firewall logs, DHCP logs, and other file logs with ingest pipelines, grok, and painless scripts
If you care about the location of an IP, geoip is built in now, and is pretty handy (The database is built into Elastic’s container - yay! but you’ll need to mess with templates to indicate the data type - boo!)
Honestly? You might want to use Logstash. There are more examples in the wild, and you can get data on the performance of each component in a pipeline. The efficiency of ingest pipelines, assuming you don’t need it, is likely less valuable than the benefits you get from using Logstash
Palantir is bad, but this repo has some very useful data about what you might care about in the world of Windows event logs. I translated some of this to a Winlogbeat config. At some point, it would be awesome if someone were ambitious enough to write an WEF XML to Winlogbeat yaml translation tool : )
You probably noticed I point Security events to a particular index name. This points us to an index lifecycle management policy. These are useful!
There’s a ton of interesting tools and blogs out there. Many can be found in security-oriented places. Roberto Rodriguez has some awesome material, including tools like HELK. Pablo Delgado has some handy materials as well
Don’t restrict yourself to built-in log sources! Be sure to consider integrating other tools like Sysmon into the flow. You might even generate your own event log entries from your various scripts and software, and pull it into Elasticsearch, where you can alert on it, report on it, etc.

That’s about it! PowerShell? Sure. You’re not defining everything in the Kibana dev tools, right? You might have a repo that has ingest pipelines, templates, index patterns, ilm policies, scripts, watchers, and other Elasticsearch or Kibana config that can be defined in code, that you test / lint before pushing to Elasticsearch or Kibana in your build. A glue language like PowerShell is great for this!

Winlogbeat and pipelines and painless scripts, oh my! was originally published by Warren F at Rambling Cookie Monster on May 18, 2019.

So You Want to Go to the Summit

2018-11-01T07:00:00+00:00

Overview

Rambling
So you want to go to the summit
- The People
  - Tips
- The Content
  - Tips

Rambling

I guess I’ve been doing this a while! I first harassed my manager about the PowerShell Summit back in 2013.

Back then, I would look at the sessions, look at my team’s gaps, projects, and goals, and point out the sessions that could help. Nothing more.

Nowadays? The breakout sessions themselves are awesome, but aren’t even in the top three reasons I give my boss.

So you want to go to the summit

I’m just going to list out some benefits to going to the summit. Depending on where you are in your career, what you already know, etc. some of these may or may not be helpful.

The People

The people you see are hands down the biggest benefit of the summit for most folks.

Who will you meet?

Folks on the PowerShell team, and some alumni
Various leaders in the community - Microsoft MVPs, how-are-they-not-MVPs, speakers, and more
Folks whose work will send them to a PowerShell + DevOps summit, or who fund it themselves

Now… Why does this matter? How can this be used to justify going to the summit? (s/they/folks you meet)

They might have a great opportunity for you at some point in your career
You might have an opportunity they’re an awesome fit for at some point
They might be an expert in some topic you need guidance on at some point
If they’re involved organizing conferences or user groups, you might get a chance to speak
They might have some ideas or conversations that give you an idea on how to solve something at work
They might be working on or know about a project / tool that would be perfect for your org
Folks can share more in person than in a blog / public. You’ll find folks from Microsoft, Amazon, and many other companies doing awesome and interesting things, who might not otherwise blog or publicize certain things

Meeting new folks doesn’t come natural to everyone. Certainly not me. What can help? How can you get the most out of this?

Tips

Get to know folks!

Join the community. Ideally before the summit. Twitter, Slack (Summit Slack and PowerShell Slack at least), Reddit, PowerShell.org, etc. Ask questions. Help others. Chances are you’ll meet folks from these communities in person, and this is a great way to keep in touch with folks after the summit. You never know when they might have a job opportunity pop up!
If you can, put your phone away at breakfast, lunch, and breaks. Say hi to new folks or folks you’ve already met
Go out to dinner and evening events with the folks you meet. Check Slack (where are folks headed tonight? will surely net you one or more choices!)
Attending with co-workers? Ditch them. Or at the very least, be sure you’re not shying away from meeting other folks simply because it’s easier to stick with a co-worker
Don’t be afraid to start the conversation. There are plenty of simple questions to start with! So where are you from?, Is this your first summit?, So what do you do?, Any fun projects you're working on?, etc.
Introduce yourself to the speakers, organizers, Microsoft folks, etc. They’re all just people. Don’t be afraid to talk to them!
Work can wait. You only get this opportunity once a year, don’t eat the most valuable time at the summit working, if you can help it
Introverted? Drain your batteries. Seriously. You can recharge after the summit. Eat lunch with folks. Walk up to new folks and start a conversation. Go out for food and other fun in the evening
Keep coming back. It gets easier every year. As you get to know folks in the online communities, and come back to the summit, it becomes increasingly easier to find folks you know. But… do meet new folks!
Never take the red eye. Thursday night gives you one last chance to catch up with folks. Some folks do a huge pub crawl. Others (me!) prefer a quiet night getting food and drinks with other summiteers

The Content

So! On top of people and hallway track that they create, you have actual summit content.

General sessions (everyone in the same room)
Breakout sessions (45 minute or 90 minute bits)
Side sessions that can range from lecture-ish bits, to group discussions, to hackathons
Lightning demos - 10 minute or less talks that move fast and introduce you to interesting things you could dive into

The schedule won’t be completely filled out before you ask for approval, but there should be enough to help.

There are a few ways you can use this to encourage your boss to send you:

Line up session topics with projects or goals at work
Line up speaker expertise with projects or goals at work - you can talk to these folks after all
If this is like the previous years, you’ll get a preview of what the PowerShell team is working on
You’ll have the chance to talk with various folks on the PowerShell team
Even if a good bit of the content is posted online, being there means you can follow up with the speakers
There’s a high signal-to-noise ratio. This isn’t some sales-y conference with vendors and parties every night. This is a group of your peers

In case it helps, here are a few tips on content…

Tips

Take notes! What did you learn? Who did you meet? What were they working on? What ideas / tools came up? etc.
The breakout and most general sessions are recorded. Unless you really want the opportunity to see the content and follow up with the speaker, you can catch these after the summit - don’t feel bad skipping them!
The hallway track and side sessions are not recorded. My most valuable takeaways are from people / side sessions every year. This is the stuff you can’t get from videos posted online after the conference. Don’t miss these!
Plan ahead! Are there any sessions you can’t miss? Any where you know you’ll want to hit the hallway/side sessions? Reduce the last minute scrambles - at least have an idea of where you’ll go, and be sure to check for updated side session topics
Look for related events! DevOps Days Seattle is a week ahead. There might be a WinOps-y nano conference the day after. Poke around and ask folks before you book your tickets and hotel!

That’s pretty much it! There’s even more you could add if you start sharing - doing lightning demos, speaking, leading side sessions, organizing, etc. - but the bits above should apply to pretty much everyone interested in the summit.

Oh! Registration opens today. Hope to see you there!

So You Want to Go to the Summit was originally published by Warren F at Rambling Cookie Monster on November 01, 2018.

The 2019 PowerShell and DevOps Global Summit CFP

2018-10-16T07:00:00+00:00

Overview

Rambling
Deciding
Some CFP Stats
How did you work on the CFP?
- Things that worked
- Things that didn’t work

Rambling

Whew! It’s been a fun and busy few weeks. I’ve been working with Missy on reviewing proposals for the 2019 PowerShell + DevOps Global Summit.

I was excited and honored when I was asked to help with the summit. I’ve been coming to this summit since I started working on PowerShell, and I owe most of any success I’ve had to the various attendees and speakers who have helped me along the years.

Working on the CFP has been awesome - you get to preview a ton of interesting ideas, and you get to help pick the content of the summit, which sets the direction of topics and ideas for a decent sized audience… It’s a huge honor, and I would do it again, but I can assure you, turning down ~150 proposals, and reviewing 20 release pipeline bits can wear you out - I’m happy to be done for the year!

Now that the agenda is out and speakers have been notified, I figure it’s a good time to talk about the CFP!

Deciding

Picking topics was tough. Cutting topics was tougher. Let’s look at some of the factors!

The Basics

Early in the process Missy pointed to a solid bit on reviewing proposals for a CFP. This helped serve as a gate of sorts.

A few of these criteria lead to some hard _no_s. We weren’t super strict, but some things scare a reviewer off. If you put no time into a proposal, don’t review it, and have no one else review it, leading to a bunch of spelling and grammar errors among other issues, we might extrapolate and think… what sort of presentation and material will this author put together?

Other criteria simply made it harder for us to say yes. We had a bunch of release pipelines, Azure, and Pester proposals. This means you have a lot more competition - not only is your proposal up against everything else, it can be edged out by proposals on similar topics. It also means your proposal might blend in with the others, and stand out less than an original topic.

Combined with other factors, the why is this important bit stood out. Proposals are basically marketing. We read through each proposal multiple times. If you didn’t capture our curiosity or interest, it made it much easier for your proposal to land in the maybe pile, which puts it at a little disadvantage. If it was on a popular topic and it didn’t intrigue us? It might have ended up in the no pile.

There was more, but those are three criteria that stood out to help as an initial filter.

Did you spend any time creating and reviewing your proposal?
Was your topic very popular?
Did you capture our interest?

So… We’ve talked about some things that lead to a no or a maybe. What made us say yes?

The Good Things

Some factors helped proposals get to the accept pile, for better or worse. Keep in mind these are simply factors that could help to some degree, not hard-and-fast rules.

Was it unique? This makes it much easier to say yes. Particularly for a popular topic. Module or infrastructure release pipelines? Tons of submissions. SQL release pipelines? One
Was it widely applicable? If your talk can extrapolate across different products / stacks, it had a better chance. For example, we might pick a bit on psake, over a similar bit specific to Azure DevOps
Do you have any community presence?. This might be a bad one. There are plenty of situations that lead to someone having no chance to contribute at work and at home. That said, your work in a community can help us pick some ‘known quantity’ speakers - whether that means you’re an awesome speaker that we’ve seen, that you’re an expert in a particular topic, or that you’re quite creative and tend to do awesome things. This could even mean just networking with folks at the summit
Was your topic interesting? On top of being unique, having an interesting topic can help. This really depends on the reviewer, and what they think the audience, or subsets thereof, might find interesting
Are you the author? This is a pretty specific one, but worth pointing out. If you’re the author or maintainer of a tool or module, chances are if there are more than one proposal on it, you’ll be speaking
Did you submit late? We’re human. We need to review and follow-up throughout this CFP process. We did our best to avoid any bias based on time, but to a small degree, I think the farther along the CFP, the more compelling a proposal needed to be to land in the accept pile
Did you get lucky? Again, we’re human. Maybe Missy or I were in a good or bad mood when we saw / re-reviewed your bit. Maybe you ended up in the maybe pile and we didn’t give your proposal as much consideration on re-review as we should have. There are so many little things that could have added up to what I call… Luck

Okay! Let’s move on to some more potential road blocks.

Things that didn’t help

Some of these will be the inverse of helpful factors, or repeats from the basics… In general, these didn’t help:

Hot topics. Proposing a popular topic? Yours needs to stand out that much more
Constrained topics. Azure is a good example. Not everyone uses Azure. If we got 20 world-beating Azure proposals from the brightest leaders in the community, we’re still only going to accept a handful or less
Lack of context or clarity. This came up in a few ways. If you’re talking about PowerShell with technology xyz, tell us (and the audience!) what technology xyz is! If you’re giving us a recipe for solving something, give us a light outline of sorts - it can help us if we’re not sure you’ll go off in an odd direction - particularly important for new folks
A single proposal. I know. It takes time to write these. But here’s the thing: What if we happen to get a blockbuster proposal that covers what you cover? What if you simply get unlucky? Having two or three proposals gives you a better chance of speaking
Repeat topics. We don’t mandate original content. We completely understand that coming up with new talks is hard, and that iterating on topics works. That said - if you’ve given this talk at a recorded conference or even at this summit another year, you need to tell us what will be different, or there’s a very high likelihood it’s going in the no pile, even if it was originally a hit
Poor fit. Did you submit a beginner level bit? Something that isn’t PowerShell or DevOps related? It probably won’t get picked. Even if we let some beginner level material through, chances are it’s going to have some serious competition, and we likely won’t let anything through that would be covered in the OnRamp material

Whew! Okay, so we’ve talked the basics, and factors that made it more or less likely for us to select your topic. Let’s cover a few other factors that you might not think about.

Logistics and balance

So! These are some other factors that could help or hurt a proposal:

Two accept-worthy proposals. We have budget for ~40 non-PowerShell.org speakers. We have 56 or so speaking slots. This means that we need several folks to give more than one presentation. If you submitted two strong proposals, chances are you got in, even if we only accepted one (we might have added your other as a potential backup). I know this isn’t optimal, but we have a cap on attendees, so every speaker means less $$ to support the summit
We need a balance of known and unknown speakers. Known speakers can help sell tickets, and we know they’ll likely give solid sessions. That said, we want to encourage folks to share and speak, so we want new folks as well!
We want to introduce more non-PowerShell-specific material. As folks progress in their careers, PowerShell will likely become less of a central focus. It will still play a large part of this conference, but we’ll be including sessions on important topics that a PowerShell-er might run into as they progress in their career, even if those topics don’t involve PowerShell itself
We want to balance content We mention it above, but again, we need to balance content. We can’t have 20 Release Pipeline talks, and 10 Azure talks

Let’s summarize these

So you want to speak at the summit

That was too many words. Let’s boil it down to some takeaways that might help you with a successful proposal:

Do submit a proposal. Seriously.
Do give it a small bit of effort. Write it. Review it. Give it to someone else to review. Adjust as needed
Be wary of over-populated topics. These can be hard to judge. You can always ask!!
Do make your proposal stand out. It needs to capture our curiosity or interest
Consider making it unique. Pester? It’s been done. Pester-for-some-unique-thing? Much more likely
Consider keeping it widely applicable. Even if you use a particular stack for your demos, let us know you’ll include widely applicable takeaways
Consider interesting topics. This depends on the reviewer and the target audience, but is worth considering
Do submit as early as possible. You’ll stand out more initially
Be wary of constrained topics. We’re not going to have a conference of all Azure or AWS bits
Do include enough information for us and your audience to know what you’ll be talking about, even if it’s a very rough outline
Do submit more than one proposal. Don’t throw everything against the wall, but don’t risk your single proposal getting unlucky
Be wary of repeat topics. At the very least, if you’ve done it before, let us know how it has changed
Do keep the topic appropriate for this conference. Not sure? Just ask!
Do get involved with the community if you can. Blog, share / open source your work, attend and speak at user groups, do a PSPowerHour, etc.
Do ask for feedback. We created #Conferences in the PowerShell Slack team. I suspect this would have helped on a number of bullets above, and for a number of proposals

Okay! Let’s talk about the CFP itself

Some CFP Stats

We have a bunch of data on the CFP proposals. Nothing particularly interesting, but if you’re curious:

We had something like 205 proposals. We actually had to send back a few ahead of the deadline, to make sure folks weren’t locked out by the 200 cap
We had 56 accepted proposals. 46 standard (45 min) sessions, 10 double (90 min) sessions
We have 21 new speakers, and 19 returning speakers. This is in relation to this specific summit. Perhaps 5 to 10 of the new speakers speak at other PowerShell-y conferences
We have something like 11 Microsoft MVPs. This doesn’t include folks like me, who aren’t speaking unless we get pulled in as a last minute backup, or the folks like Don who are speaking only at the OnRamp track. Also, the whole MVP thing? It can be nonsense. We have folks like Brandon Olin and Mathias Jessen who… aren’t… MVPs. Right
We have 40 total unique speakers. Only one is from PowerShell.org (James Petty). I’d much rather listen to you all speak, than to share myself, even if you took the I-get-nervous-before-speaking and it-takes-time-to-prepare-a-talk factors our of the equation
75 unique speakers didn’t make it. This was seriously painful. I know many of them from the community, and have met a good number of them in person. I encouraged many of them to propose. This was the hardest part of the CFP for me
10 Microsoft MVPs didn’t make it, along with several Microsoft folks, and a number of other awesome folks in the community
There’s some overlap given that proposals can cover more than one topic, but looking at a boiled down what was the primary topic, the three most popular topics were:
- Release Pipeline - 20 proposals
- Azure proposals - 13 proposals
- Pester proposals - 11 proposals
We had some prolific proposers. I can’t imagine coming up with 9 ideas and enough to propose on each. Two speakers did

How did you work on the CFP?

So! This was my first time being responsible (well, co-responsible!) for a CFP. Some notes for posterity:

Things that worked

Google Sheets. PaperCall and other services might not have the best reviewing tools. Some easy-to-collaborate spreadsheet service works for this
Having help. Don’t go alone. Seriously. IMHO no CFP should be handled by a single person. The stress, items you might miss without a second pair of eyes, and other issues will pile up. Thanks Missy : D
Knowing spreadsheet language basics. Using spreadsheets? Learn the language (functions, formulas, etc.). There’s only so much manual work you can do. Sheets and Excel both have useful query languages, and you should know enough to pick these up without too much trouble
Keeping identifier data to tie review data to CFP data. Make sure each review entry has enough data to uniquely tie it back to a talk - for example, the submitter’s name or e-mail and the title of a talk. Watch out for whitespace. This helps reconcile the data if you need to join it up later on
Using PowerShell! Once we had review data in sheets, and proposal data in PaperCall, it was easy to join, filter, etc., generate speaker agreement word documents, and send acceptance, back-up, and not-selected e-mails via smtp. I also had a quick way to generate all the speaker’s twitter handles, stats for this article, etc.
Encouraging people. I can’t tell you how many folks I pinged, who either hadn’t considered proposing, figured they didn’t have anything worth proposing, just didn’t see that we were running a CFP, etc. A number of these folks were selected. I’ll remind folks again next year, but seriously, if something helps you or your teammates, chances are it might help folks at this conference. Propose it!
- Side note: if I missed pinging you, I’m sorry! tried to get a bunch of folks, but I missed plenty
Remote collaboration tools. Already mentioned Google Sheets. Slack and screen-sharing-tools (e.g. Skype) also came in quite handy when working with other reviewers

Things that didn’t work

Manual work. Each time a new proposal came in, we manually copied data over to our review sheet. This was painful
- Suggestion: have an easy process to add proposal metadata to your review data, without messing with existing review info. For example, download your review data (e.g. sheets), download your PaperCall data, join it, and publish the data back to Sheets. Maybe use existing tools
Not linking review data to CFP data. We used google sheets for review. Each time we needed to review proposal content, we needed to browse to PaperCall, search (its search tool is horrible), select, and scroll to the content
- Suggestion: Make it easy to tie your review metadata and notes to the CFP data. A link to the CFP proposal for each row in your review, for example
Too many Maybes. We each rated a talk as it came in. Yes, No, Maybe. I think we (I did, at least) might have put too many in the Maybe pile. We revisited the Maybe pile multiple times, but not too many proposals went from Maybe to Yes, and I worry that we missed some gems
- Suggestion: Be as thorough as possible when you first rate a proposal. Don’t look at a proposal if you don’t have time to review it in full. Try to avoid marking proposals Maybe unless you really mean it.
Tags, not buckets. We tried to boil down each talk into a bucket (e.g. Release Pipeline, Azure, Serverless). Here’s the thing: Many talks might have two or more topics involved, and might include a constrained topic. For example, Release Pipelines overlapped heavily with Infrastructure-As-Code, and the Serverless bucket often included a constrained topic (Azure or AWS)
- Tag central topics for each session. Submitters often do this, but they might be overzealous, or use different words - normalize and filter these. Spreadsheets aren’t the best for many-to-many relationships, but you can shoehorn it in

I’m sure there’s more, but this is mostly a quick brain dump for next year’s CFP

Whew! That’s it! Tickets go on sale November 1, and you can find a brochure, agenda, and other links here. The initial agenda looks fantastic (I know, I’m biased). Hope to see you at the summit!

The 2019 PowerShell and DevOps Global Summit CFP was originally published by Warren F at Rambling Cookie Monster on October 16, 2018.

Quick hit: PoshBot Useability

2018-09-13T07:00:00+00:00

Overview

Rambling
Isn’t PoshBot Just PowerShell?
No Tab Completion
No Pipeline
- Do everything in your command
- Format things for the user
Output Needs More Consideration
- Pick sane defaults
- Don’t spam
That’s a Whole Bunch of Work!
- Is there an easy button?
- You’re wrong!

Rambling

So! First post in a year. Guess setting up the Quick Hits section was totally worth it :)

It’s been a great year! On top of various fun with PowerShell, keeping up with a two-year-old destroyer-of-all-the-things, and having a new destroyer-of-all-the-things in training, I’ve spent a good bit of time exposing handy tools in Slack via PoshBot, an awesome bot framework by Brandon Olin

We’ve been using PoshBot for over a year (see: last post), and we finally have Microsoft Teams support in PoshBot - figured I’d try to lay out some practices and tooling I’ve found helpful in case it would help anyone!

Isn’t PoshBot Just PowerShell?

Nope. Not. At. All. Let’s start out with what we don’t have:

No tab completion. Quick discovery and command/parameter validation simply isn’t a thing
No pipeline. Yet
Output needs more consideration. Unless you want to make your rooms unreadable

So! What does this mean? How can we make this more usable?

No Tab Completion

Get ready for co-workers who can’t remember, or mistype commands and parameters all the time. Heck, I write most of our plugins, and do this all the time. What can we do about it?

Use aliases

How do I force AD replication? !syncad? !adsync? Something else? Here’s the thing: aliases are free. If you see a very common mistyped variant, and if it makes sense and won’t be confusing now or when you add more commands, go ahead and add it!

Keep it short

Verbosity is king in PowerShell. With no tab completion to help discovery and spelling? Keep it short! Reserve single character aliases for your most popular commands and parameters. Expand as needed. Again, you can always leave the actual command name in a PowerShell Verb-Noun format, but do use short aliases for chat.

Don’t forget parameters! Want to specify properties? -p is a nice shorthand for -property, for example.

Keep it consistent

This goes for PowerShell as well. If you have a parameter that offers the same functionality in many commands, be sure you also give it the same name and aliases in every command. With no tab completion, this simplifies memorization, and lets us extrapolate syntax and naming from other commands.

Make it easy

A busy command definition is much nicer than forcing a user to run a busy command. Abstract things out. For example…

You want to query users from Active Directory. What are some common criteria? Not everyone will want to type raw ldap queries in chat. Can you inspect parameter values and make assumptions (clarifying in the help for each parameter)? For example:

!u wframe: Get users with identity wframe
!u wframe@g.something.edu: Get users with mail wframe@g.something.edu
!u 12345: Get users by UID
!u 'warren frame': There’s a space! Use ambiguous name resolution

Positional parameters are also quite helpful, for your most commonly used parameters.

Give good examples

Comment based help examples should already be the norm, but be sure to do a solid job showcasing these - I tend to use full command/parameter names in the first example, and the easiest shorthand thereafter. Reading help is a bit more painful in chat. Examples give you a quick way to say ah, this is exactly what I need to do!

No Pipeline

Do everything in your command

So! Forget the Monad Manifesto and PowerShell best practices for now. You can’t pipe. Want specific properties back? Include a -Property (-p) parameter. Want to format data as a table, list, csv, or something else? Include a -Format (-f) parameter. Think about other common needs.

Once you gather up a common group of parameters, strongly consider adding them to every PohBot command they apply to - no output on a command? Yeah, probably don’t need -Format (-f) on that one.

Format things for the user

So! We just mentioned this. We can’t just pipe to Format-Table or Export-Csv in chat. You probably want some common format options via -Format (-f). Tables. Lists. CSVs. Pretty spreadsheets with tables. No one wants a command that spams a room with unnecessary lines of text or whitespace - give folks only what they need.

Be opinionated! For example…

Did they use wildcards or some other fuzzy search and get a bunch of results? Bundle it into a CSV, or into a table with only the properties they would need to get the details to query further (e.g. userprincipalname, displayname, mail for an AD user)
Is the text-based representation of the output longer than you want in the channel (e.g. > 8000 characters)? Switch Format over to something like a CSV, which Slack does not truncate
Is the output a single item with a single property on it? Expand it! Do include the property name to help tie the output back to the command
Is there something abnormal in the output that should be highlighted? Consider doing so! Are there disabled users in your output? Consider listing them out (alphabetically of course) in a warning, ahead of the command output
Is there output that generally isn’t needed? Consider excluding it by default, and adding a switch to override - (e.g. AD group membership data might exclude disabled users by default)

Output Needs More Consideration

Pick sane defaults

Get-ADUser gives me back DistinguishedName, Enabled, GivenName, Name, ObjectClass, ObjectGUID, SamAccountName, SID, Surname, and UserPrincipalName. Different folks have different needs, but I’m guessing most of us don’t care about most of those properties. Including them in chat, with it’s limited character width and ugly wrapping behavior, would be painful.

Pick sane properties to return by default, and allow the user to specify more properties (ideally, support -Property * to allow discovery). Pick a sane property to sort on by default, if that makes sense. Pick a default format (e.g. table, csv, list) based on the most common use of a command.

Be sure to consider your chat system’s line wrapping. Using Slack message attachments (common)? You get ~78 characters. Can you get away with a table? Will some data make your output unreadable? You can always adjust things as you see issues pop up in chat.

Don’t spam

Just don’t. Already mentioned this, but be sure to inspect your output. Recent PSSlack changes added support to handle rate limiting, but still, you don’t want to destroy the usability of whatever channel you’re chatting in!

There are many ways to do this. I tend to globally say if the string output is longer than 8000 characters, you get a CSV or whatever file type is appropriate. Slack truncates messages. Larger items fit much more nicely in a file

Also, be careful with errors. PoshBot will pick up errors and send them to chat. Any intentional manipulation of output you do is completely ignored at that point. Let me get this big list of group members! Oops. Formatting error on each of the 100 members that came back. Slightly embarrassing : )

That’s a Whole Bunch of Work!

Perhaps. Most of this is just thinking about how folks will use the tools you write, and accommodating a platform that doesn’t come with the niceties of a CLI.

Just consider: If you don’t spend a little time thinking about this, you might lose out on the benefits of ChatOps - folks might start simply using direct messages instead of actually working in-line with chat. Some folks might get frustrated if you have inconsistent parameter names, or only a single command name that they forget every time.

Also… I don’t know about you, but I love convenience. Much of this is bad practice in PowerShell, given that we can use everything in the pipeline, but here you can get as opinionated as you’d like, while still doing the right thing and making life easier for you

Is there an easy button?

Not really. Most of this will become apparent as you use PoshBot. Just start using PoshBot. See what works. Ask around to see what you can improve. Don’t turn your nose up at ideas that fly in the face of PowerShell best practices - ChatOps is a different context.

I’ve written a module that can make things a little easier, but it’s ugly, and might not fit your needs.

I tend to write all of my functions with a set of their own common parameters like -Format or -Property. I then pass any output from a PoshBot command to ConvertTo-PoshBotResponse, a function that knows how to parse these common parameters, checks for lengthy output and switching to CSV, and other niceties.

I’ll publish the code and write a short bit on this soon!

You’re wrong!

What do you think? Do these practices make sense for bot commands? Do you have other good practices to follow? Are any of my suggestions misguided? I’ll try to include a summation in my next post!

That’s it for today! It’s been a while so I’ll mention a few things I think you should check out:

Michael Lombardi and I recently started PSPowerHour. Basically, 8 people do lightning demos on something fun. It’s a great, low-pressure way to show off something fun and get experience speaking. Do submit a proposal!
I’m working with Missy Januszko on content for the 2019 PowerShell + DevOps Global Summit - We have some awesome sessions lined up, but need more - do consider submitting a proposal!
Mike Robbins put together an amazing PowerShell book for a good cause. It’s available on leanpub and now Amazon - do check this out!

Quick hit: PoshBot Useability was originally published by Warren F at Rambling Cookie Monster on September 13, 2018.

Getting Started with PoshBot

2017-06-25T07:00:00+00:00

Overview

Rambling
Why ChatOps? Why Bots?
PoshBot
What’s Next?
- We Don’t Trust Bots
- References
  - What about PoshBot itself?

Rambling

One of my favorite things about events like the PowerShell + DevOps Global Summit is the opportunity to chat with so many folks in the PowerShell community. Invariably, you end up bringing back a wealth of fun ideas.

This year, Brandon Olin gave an awesome lightning demo on PoshBot, a PowerShell-based bot framework. It’s not quite as mature as Errbot or Hubot, but it’s a super quick way to get a bot up and running for folks using PowerShell!

Why ChatOps? Why Bots?

I’ll let someone more well versed in the topic handle this - Jason Hand has a great beginners guide to ChatOps. Long story short?

Communication - avoid certain folks monopolizing discussions, allow quiet folks to get important details out, keep everything in a searchable medium for new and existing team members
Transparency - information isn’t hidden in e-mail threads that new or excluded folks will never see - everything is searchable, and ideally you do work in open channels
Sharing - tools and ideas can be shared in a single spot
Learning - find details on old topics that never made it into docs, see how people solve things with bots
Automated communication - changes to code, deployments, alerts, etc. can be sent to chat
Security - Guardrails and other safeguards can reduce risk (consider what JEA provides)
Incident response - keep incident communication in central, searchable location, not a bunch of incomprehensible e-mail threads
Provide an interesting balance between creating a web interface or GUI for a PowerShell script, and forcing folks to use PowerShell
Share gifs

I could be off base, but I also see this as a prerequisite for effective remote work.

So! Hopefully you’re interested in checking this out. It helps if your team is already using something like Slack.

Let’s dive in and start using PoshBot!

PoshBot

We’re going to cover the basics to get up and running with PoshBot:

Create a Slack bot
Create a PoshBot configuration
Run PoshBot as a service
Write a PoshBot plugin
Use PoshBot

This might seem like a lot of work, but the configuration and service are a one time thing - Writing plugins is just like writing PowerShell functions and modules!

Create a Slack bot user

This assumes you have admin privileges to your Slack team:

Add a new bot integration: https://YOUR-TEAM-HERE.slack.com/services/new/bot
Name the bot - we’ll use testbot
Optionally, pick an icon
Copy the resulting token for the bot config. Keep this safe!

We have our token, let’s get a bot configured!

Configuring PoshBot

There’s more you can configure, but we’ll start with the basics. Keep in mind you might abstract this into your configuration management system, use infrastructure as code, etc.

That’s it! We can quickly test this interactively: Start-PoshBot -Configuration $pbc -Verbose

We should see the bot load up and start listening:

VERBOSE: Creating new Slack backend instance
VERBOSE: Creating bot instance with name [testbot]
...
VERBOSE: [SlackBackend:LoadUsers] Adding user: REDACTED:wframe
...
VERBOSE: [Group: AddUser] Adding user [REDACTED] to [Admin]

If we look in Slack, we’ll see the bot in the list of users, but we still need to invite it to any channel we want to use it in - /invite @testbot. You can also DM the bot.

That’s it, we have a bot! Let’s set it up to run as a service.

Run PoshBot as a service

There are several ways to run PowerShell as a service; We’re going to use nssm - it’s a bit simpler than some of the alternatives.

First things first - we need the script that will run as a service - C:\poshbot\start-poshbot.ps1. This is pretty bare bones, you could do more:

Basically, read the config, run Start-Poshbot. Forever. Okay! Let’s run this as a service:

If all went well, in a few seconds we should see testbot online in our Slack team again!

Now for the fun part. Let’s write a quick PoshBot plugin!

Write a PoshBot plugin

A PoshBot plugin is basically a PowerShell module with a few PoshBot bits. We’ll skip the typical module scaffolding to keep our example simple.

We’ll create our poshbot.example module in our plugins directory:

$null = mkdir C:\poshbot\plugins\poshbot.example -force

Next, we’ll create a quick module manifest, Poshbot.Example.psd1:

The key bits are to require PoshBot, and while not mandatory, describing the permissions can be helpful.

Finally, we’ll draft a quick module script, Poshbot.Example.psm1:

Whew!

The key bit here is the [PoshBot.BotCommand()] custom attribute - we use this to tell PoshBot what the command name is (if different from function name), permissions required to run this, and aliases that will also call this command.

So, that !user command is pretty ugly. Unfortunately, we’re no longer in PowerShell land, where we write commands that do one thing and do it well, using the pipeline to connect things together.

Some things you might consider:

Add different format options. A table is pretty and easy to digest, but won’t always cut it
Provide PowerShell-esque Verb-Noun names, but add aliases for everything. Short commands and parameter names are important, when you don’t have tab completion or intellisense
If it makes sense, add the ability to specify what properties to output, and pick a sane default
If it makes sense, consider writing a file and uploading it to Slack (e.g. a graph png, a spreadsheet), or linking to external URLs
Publish your module to the PowerShell gallery with a name starting Poshbot. - PoshBot will automatically find these when you use !Find-Plugin

We’ve written a PoshBot plugin - how do we use it?

Using PoshBot

So! We have a bot, it’s running as a service, and we’ve started writing some custom commands - let’s play!

install-plugin poshbot.example

!user wframe

Oh. We have to deal with access controls first!

PoshBot Access Controls

We’ll create a group and role, tie the group to the role, tie permissions to the role, and add users to the group:

!New-Group sysadmins
!New-Role sysadmins
!Add-GroupRole sysadmins sysadmins

!Add-RolePermission sysadmins poshbot.example:read
!Add-GroupUser sysadmins wframe

And finally: !user wframe

Getting Help

How did we know how to do that? Mostly reading docs, but you can also use the !help command if you ever forget a command name or syntax:

!help

!help example:user

Note that we need to specify enough of a command for it to be unique before we get the full help, we can’t just use user.

Running Commands

We’re ready to go! Some quick examples using our functions:

Use an arbitrary LDAP filter, and specify some specific properties to return: !user --l '(mail=wframe*)' --p samaccountname, displayname, enabled

Check out what variables are available when in a PoshBot function: !var

That’s about it! Now you know how to start up your own PoshBot in Slack, and write custom commands.

What’s Next?

Many folks start with read-only commands until they get comfortable with things. Some example tasks you might write commands for:

Get stuff from AD - groups, group membership, users, etc.
Do stuff in AD - disable or unlock accounts, kick off replication
Get stuff and do stuff from a variety of other systems!
- Tickets
- Configuration management systems
- CMDBs or other inventory sources
- IPAM systems
- Monitoring systems
- Databases
- etc.
Restart services and servers
Kill stuck sessions
Query log data, performance data, etc. - this can be text, graph based, anything else (you can create and upload arbitrary files!)
Provide quick shortcuts to common docs

For all of these, you’ll need to consider a way to provide access to your service account. Some common approaches:

If possible, use a password manager, or serialize credentials via DPAPI so that your service account can read them and use them in commands
You might create delegated, constrained endpoints, or use JEA
You might run the service as an account that has delegated access (be extra cautious here), etc.

Consider read-only accounts, or delegating only the privileges needed for any approach you take. Also, you might classify your system appropriately to ensure appropriate security controls are in place : )

Unfortunately, some folks are apprehensive about bots. What if something breaks? What if someone breaks in?

We Don’t Trust Bots

I’ve heard this. I’m sure others will hear it as well. Boiled down, PoshBot is pretty secure:

There’s no opportunity for injection. PoshBot doesn’t execute text, it uses the AST to validate commands and parameters
Only commands PoshBot knows about can run
PoshBot enforces access controls via groups, roles, and permissions
If an author considers what could go wrong, guardrails and other safeguards can make a command more safe than running natively - think about the benefits of using JEA. Maybe I provide access to restart services, but only a subset of services that make sense

There are still a few risks. What if Slack itself is compromised? What if an account is compromised? A few things to consider that would mitigate this a bit:

Use two-factor auth
Limit the extent of what your commands can do. Maybe don’t write delete datacenter, or allow add innocent-looking-backup-account to domain admins

Keep in mind security is about risk management, otherwise we would throw all our equipment into the sea and be done with computing. For most, the value of ChatOps will far outweigh the risks.

References

There’s plenty of other reading material out there on ChatOps, consider checking these out:

What about PoshBot itself?

Poshbot is quite new, compared to bots like Hubot or Errbot - there’s not much out there yet!

Brandon is doing an awesome job, but help is always welcome - found a bug? Want to help with docs? Have a feature request? Want to learn Pester and write some tests? Open an issue or a pull request!

Cheers!

Getting Started with PoshBot was originally published by Warren F at Rambling Cookie Monster on June 25, 2017.

Quick Hit: Community Lightning Demos

2017-03-20T07:00:00+00:00

Overview

Rambling
Lightning Demos
- Community Lightning Demos
- Sign me up!

Rambling

Picking between traditional sessions at conferences can be a pain. What if the content, speaker, or topic don’t quite live up to your expectations? Now you get to sit through the rest, or try to awkwardly sneak out and catch only a portion of another session.

On the other side of the fence, some folks might be scared off from speaking at a conference. 45+ minutes of material can take time to come up with. Demo gods require sacrifices. A full session can seem overwhelming, particularly if this is your first time presenting.

A fun non-traditional option is to do a round of lightning demos. Thankfully, the folks at the 2017 PowerShell + DevOps Summit accepted a proposal for community lightning demos, which I’ll be honored to host!

Lightning Demos

For the past few years, the PowerShell team lightning demos have been a consistent highlight of the NA PowerShell Summit. It’s a fast-paced session where team members plug in a laptop, and walk through a quick demo of something cool they’re working on.

This is a great format:

As an attendee, you get a variety of content, speakers, and topics; even if one or two bits bomb, chances are you’ll have some awesome takeaways
As a presenter, you only need to come up with 5-10 minutes of content; you might not be quite so overwhelmed if you’re only up there for a few minutes, and it can be comforting knowing that a number of your peers will be joining you

So! Given how mainstream and important open source projects are becoming, a round of community lightning demos could be quite fun and enlightening. What should you show?

Community Lightning Demos

This is all up to you, but here’s a basic recipe that should be easy to follow, and help the audience fill in the bits they need to know:

Pick a topic. Demo a module or function that you wrote, or that you use often. Describe a helpful tip or trick that you think is worth sharing with the audience. This is up to you!
Put together a quick demo under 10 minutes. Don’t try to stretch things to hit 10 minutes; if you say what you need in 5 minutes, even better. If things don’t go as planned and you end up hitting the time limit, folks can catch you after
Include info the audience will find helpful:
- What are you talking about?
- How does it work? This is a demo after all!
- Why might the audience use this?
- Where can they find more?

That’s it!

Here’s a quick example lightning demo walking through PSDepend basics. Sorry about the volume, there was a sleeping toddler in the other room…

Sign me up!

You can sign up here - more details at the link.

No need to spend too much time prepping for a few minutes, but do consider:

Putting together a demo. Mostly code and working examples
Running through your demo, including what you think you’ll say
Testing your demo again, if you update PowerShell, modules you use, etc.
Thinking of the audience. Can they read your editor? Is there syntax highlighting? Can you split lines up on pipes and other natural spots to avoid scrolling horizontally when your editor is zoomed in a bit?

That’s pretty much it, I’ll be looking forward to some awesome demos!

Thumbnail image source

Quick Hit: Community Lightning Demos was originally published by Warren F at Rambling Cookie Monster on March 20, 2017.

Quick Hit: Fun with Sorting

2017-02-24T07:00:00+00:00

Rambling

So! I’m updating a squid file to remove some duplicate URLs that break things, when my mild OCD gets the better of me. These damn URLs are not in order!

Rather than resorting to a few GNU tools, I realized I could use PowerShell from my Mac.

Sorting URLs

We’ll use the following URLs in our examples:

$URLs = 'dept55.host.tld',
        'dept1.host.tld',
        'some-other-host.net',
        'a.dept55.host.tld',
        'first.com'

We’ll try the obvious Sort-Object first:

$urls | Sort-Object

# a.dept55.host.tld
# dept1.host.tld
# dept55.host.tld
# first.com
# some-other-host.net

Nope nope nope. I want my dept55 subdomains grouped together. Also, first.com should come before anything host.tld.

It looks like we need to sort by pieces of this string. How do we do that?

Custom Sorting

Calculated Properties

You might be familiar with calculated properties. The syntax is ugly, but they’re incredibly helpful, and we can always use snippets!

New-IseSnippet -Title "Calculated Property" -description "Create a calculated property" -text '@{ label = ""; expression = {} }' -Author Blah -CaretOffset 12 -ErrorAction SilentlyContinue -force

Shorthand, we can just use @{ l=@{}; e=@{} }.

That’s a bit ugly, but let’s show what this can do:

$Example = [pscustomobject]@{
    One = 1
    Two = 2
}

$Example

# One Two
# --- ---
#   1   2

$Example | Select-Object *, @{l='Five';e={5}}, @{l='Multiple';e={$_.Two * 2}}

# One Two Five Multiple
# --- --- ---- --------
#  1   2    5        4

Pretty cool! We can add properties to an object - they can be static (Five), or can use logic along with other properties of the object (Multiple).

Do keep in mind this kills the fidelity of the object. You get the properties, but none of the methods on the resulting selected object. If you need to keep your object pristine, you can use Add-Member:

Add-Member -InputObject $Example -Type NoteProperty -Name Five -Value 5
Add-Member -InputObject $Example -Type NoteProperty -Name Multiple -Value ($Example.Two * 2)

$Example

# One Two Five Multiple
# --- --- ---- --------
#  1   2    5        4

Wait… weren’t we talking about sorting?

Custom sorting

So! With a calculated property, we provided a name and expression. When sorting, all we need is the expression scriptblock:

$Urls | Sort-Object @{ e={ ($_ -split '\.')[-2] } }

# first.com
# a.dept55.host.tld
# dept55.host.tld
# dept1.host.tld
# some-other-host.net

That’s a start! What happened? We basically said to sort by…

Splitting the items on .. Split uses regex, so we escape the ., hence \.
Picking the second to last item in the array ([-2]): we don’t want to sort on TLDs.

Hmm. You might have noticed that even though host.tld is sorted behind first.com, dept55 is in front of dept1. That doesn’t look right!

So, we need to sort on several components, not just the host name:

$Urls | Sort-Object @{e={ ($_ -split '\.')[-2] }}, @{e={ ($_ -split '\.')[-3] }}

# first.com
# dept1.host.tld
# dept55.host.tld
# a.dept55.host.tld
# some-other-host.net

We’re on the way! We sort by the second to last item, and then third to last item.

Refactoring

Normally, I would use the code above. It’s more readable, even if it repeats code. That said, let’s illustrate a re-usable means to handle some sorting scenarios:

# Go a few layers deep
$URLs | Foreach-Object {
    $Array = $_ -split '\.'
    [pscustomobject]@{
        Item = $_
        2 = $Array[-2]
        3 = $Array[-3]
        4 = $Array[-4]
        5 = $Array[-5]
    }
} | Sort-Object {$_.2}, {$_.3}, {$_.4}, {$_.5} |
    Select -ExpandProperty Item

# first.com
# dept1.host.tld
# dept55.host.tld
# a.dept55.host.tld
# some-other-host.net

Notice we didn’t use the @{e={}} syntax here. Turns out this is unnecessary, all we need is a scriptblock!

What happened here?

We break the URL into an array of strings
We create a new object that has the original item to sort, along with pieces of the array to sort on
We sort on the pieces of the array
We extract the item we wanted to sort

So! Even if this isn’t the best code to use for this particular situation, do keep the gist of this in mind, it can come in handy:

Create an object with the item to sort, and properties to sort on. Those properties might involve further logic and query results.
Sort on the properties in question
Extract the item you’re sorting

Other sorting options

As with any language, there are many ways to accomplish a task. Some folks will assemble a string that helps them sort. Others will use type coercion and native sorting for those types (cool example from Tim Curwick).

While all this may seem complicated, be happy that we don’t need to write our own sorting algorithms! Unless you’re a masochist that is.

Quick Hit: Fun with Sorting was originally published by Warren F at Rambling Cookie Monster on February 24, 2017.

PSDepend: PowerShell Dependencies

2017-01-06T07:00:00+00:00

Overview

Rambling
PSDepend
Getting Started with PSDepend
- What commands can I run?
- Example: Module Based Virtual Environments
  - Demo
Next Steps

Rambling

The first weeks of August 2016 were great. I spent most of my time relaxing on the porch, with some sailing, swimming, and ice cream mixed in.

Of course, I couldn’t help myself, and spent a little time on a fun side project: PSDepend

PSDepend

What is this PSDepend thing? Long story short, it’s a way for you to tell PowerShell that you need certain PowerShell modules, git repositories, and other dependencies, using a small requirements.psd1 file.

What’s a requirements.psd1?

Here’s a quick example to illustrate the basics of a requirements.psd1:

@{
    psake        = 'latest'
    Pester       = 'latest'
    BuildHelpers = '0.0.20'
    PSDeploy     = '0.1.21'

    'RamblingCookieMonster/PowerShell' = 'master'
}

This is pretty straightforward: from the PowerShell Gallery, we want the latest Pester and psake, and specific versions of BuildHelpers and PSDeploy. We also want to download a GitHub repo’s master branch.

I can now run Invoke-PSDepend against this requirements file, and pull in my dependencies.

Before we dive in, why would we want something like this?

Why PSDepend?

You can probably come up with more creative scenarios, but here are a few use cases that came to mind:

Provide functional dependency documentation for a project
Enable something like a module-focused virtual environment for PowerShell
Simplify sharing code with non-PowerShell-savvy folks
Help ensure a consistent runtime environment and improve portability for PowerShell solutions
Quickly set up a fresh dev/test/etc. PowerShell environments

How many times have you had to describe the steps to resolve prerequisites for a function, module, script, or other PowerShell code? Did you have to start from ground zero and explain what a module is and how to install it?

It turns out, many other languages have solved this problem. In Python, you might pip install -r requirements.txt. In Ruby, you might bundle install. In PowerShell? You hard code solution-specific logic into your code, abstract this out into configuration management, or just manually install the prerequisites and hope you don’t forget about them if you re-deploy your system.

Shouldn’t something like this already exist? Michael Willis has already started on the awesome looking PSRequire - unfortunately, I still need to support PowerShell pre-v5, so the class-based PSRequire wasn’t for me; time for a side project!

What dependencies does PSDepend support?

At the time of writing, we support a few dependency types:

PSGalleryModule: Modules from the PowerShell Gallery, via PowerShellGet
Package: Install a package using the PackageManagement module
PSGalleryNuget: Modules from the PowerShell Gallery, without the need of PowerShellGet
Git: Clone a repo and checkout a branch/commit/tag
GitHub: Download a specific branch or commit from a repo on GitHub, and extract a PowerShell module if we find it (Thanks to Doug Finke for some starter code!)
FileDownload: Download a file
FileSystem: Copy a file or folder
Task: Run a pre-defined PowerShell script
Command: Run a PowerShell command

More to come - Nuget for example - but we’d love ideas or pull requests! You can always run Get-PSDependType to view available dependency types.

That syntax seems too simplistic

It probably is! That being said, the simple syntax we showed with a ModuleName = 'Version' and ’Account/Repo’ = 'Version' is just a shortcut. We also support a more flexible syntax:

@{
    PSDeploy_0_1_21 = @{
        DependencyType = 'PSGalleryNuget'
        Name = 'PSDeploy'
        Version = '0.1.21'
        Target = "C:\ProjectX"
        Tags = 'prod'
        DependsOn = 'BuildHelpers'
        AddToPath = $True
        PostScripts = 'C:\SomeScripts.ps1'
    }

    # You can still mix in simple syntax
    BuildHelpers = '0.0.20'
}

Whew! That was a bit convoluted. What does it actually do?

We make sure to install the BuildHelpers dependency first (DependsOn)
We download PSDeploy, version 0.1.21, from the PowerShell Gallery (Name, Version, DependencyType)
We use a nuget.exe (and grab it for you if you don’t have it), rather than depend on PowerShellGet (PSGalleryNuget DependencyType)
We install this to C:\ProjectX\PSDeploy (Target)
We only run the PSDepend install if Invoke-PSDepend is called with -Tags prod (Tags)
We add C:\ProjectX (Target) to $ENV:PSModulePath (AddToPath)
We run C:\SomeScripts.ps1 after we’ve downloaded PSDeploy (PostScripts)

You can run Get-Help Get-Dependency, or Get-Help about_PSDepend_Definitions for details on standard attributes, and Get-PSDependType SomeTypeName -ShowHelp to see the help for a specific dependency type.

Let’s dive in and start using PSDepend.

Getting Started with PSDepend

Getting up and running with PSDepend is simple!

# PowerShell 5 or PowerShellGet installed:
Install-Module PSDepend

# PowerShell 3 or 4, curl|bash bootstrap. Read before running something like this...
Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://raw.github.com/ramblingcookiemonster/PSDepend/Examples/Install-PSDepend.ps1')

# Git
    # Download the repository
    # Unblock the zip
    # Extract the PSDepend folder to a module path (e.g. $env:USERPROFILE\Documents\WindowsPowerShell\Modules\)

# Import and start exploring
Import-Module PSDepend
Get-Command -Module PSDepend
Get-Help about_PSDepend

That’s it! Keep in mind we bootstrap nuget.exe when you import the module for the first time, unless you have it in your path or adjust the config file.

Now that you have PSDepend, we can start kicking the tires.

What commands can I run?

PSDepend includes commands to do things, and to help explore PSDepend:

Get-PSDependType

This is a quick way to see what dependency types are available:

Get-PSDependType

DependencyType  Description               DependencyScript
--------------  -----------               ----------------
FileDownload    Download a file           C:\...PSDependScripts\FileDownload.ps1
FileSystem      Copy a file or folder     C:\...PSDependScripts\FileSystem.ps1
Git             Clone a git repository    C:\...PSDependScripts\Git.ps1
PSGalleryModule Install a PowerShell m... C:\...PSDependScripts\PSGalleryModule.ps1
PSGalleryNuget  Install a PowerShell m... C:\...PSDependScripts\PSGalleryNuget.ps1
Task            Support dependencies b... C:\...PSDependScripts\Task.ps1
...

You can also use it to get help information for one of these types:

Get-PSDependType PSGalleryModule -ShowHelp

...
SYNOPSIS
    Installs a module from a PowerShell repository like the PowerShell Gallery.

SYNTAX
    C:\Users\wframe\Documents\GitHub\PSDepend\PSDepend\PSDependScripts\PSGalleryModule.ps1 [[-Dependency] <PSObject[]>] [[-Repository]
    <String>] [-Force] [-Import] [<CommonParameters>]

DESCRIPTION
    Installs a module from a PowerShell repository like the PowerShell Gallery.

    Relevant Dependency metadata:
        Name: The name for this module
        Version: Used to identify existing installs meeting this criteria, and as RequiredVersion for installation.  Defaults to 'latest'
        Target: Used as 'Scope' for Install-Module.  If this is a path, we use Save-Module with this path.  Defaults to 'AllUsers'
        AddToPath: If target is used as a path, prepend that path to ENV:PSModulePath
...

What else can we do?

Get-Dependency

If you want to parse a requirements.psd1 file, you can run Get-Dependency.

By default this will look for a requirements.psd1 or a *.depend.psd1 file in the current path. You can also use -Recurse, or specify a -Path. Here’s the output from our flexible psd1 example above:

Get-Dependency | Select-Object -Property *

DependencyFile  : C:\temp\requirements.psd1
DependencyName  : BuildHelpers
DependencyType  : PSGalleryModule
Name            : BuildHelpers
Version         : 0.0.20
Parameters      :
Source          :
Target          :
AddToPath       :
Tags            :
DependsOn       :
PreScripts      :
PostScripts     :
PSDependOptions :
Raw             :

DependencyFile  : C:\temp\requirements.psd1
DependencyName  : PSDeploy_0_1_21
DependencyType  : PSGalleryNuget
Name            : PSDeploy
Version         : 0.1.21
Parameters      :
Source          :
Target          : C:\ProjectX
AddToPath       : True
Tags            : prod
DependsOn       : BuildHelpers
PreScripts      :
PostScripts     : C:\SomeScripts.ps1
PSDependOptions :
Raw             : {Tags, Version, AddToPath...}

Notice that BuildHelpers is listed first (PSDeploy_0_1_21 DependsOn it). You can use Get-Dependency to verify that PSDepend will read your requirements.psd1 file as expected.

So! We know how to read a requirements file, but how do we actually install these dependencies?

Invoke-PSDepend

Invoke-PSDepend can kick off a test, install, or import of a dependency. By default, it will search recursively for any requirements.psd1 file, or *.depend.psd1 file under the current path, similar to Invoke-Pester or Invoke-PSDeploy, and run the install action.

Using the same requirements.psd1, let’s verify that our dependencies aren’t already in place:

Invoke-PSDepend C:\temp\requirements.psd1 -Test |
    Select-Object -Property Dependency*

DependencyFile            DependencyName  DependencyType  DependencyExists
--------------            --------------  --------------  ----------------
C:\temp\requirements.psd1 BuildHelpers    PSGalleryModule            False
C:\temp\requirements.psd1 PSDeploy_0_1_21 PSGalleryNuget             False

Invoke-PSDepend C:\temp\requirements.psd1 -Test -Quiet

False

Now, let’s kick off PSDepend:

Invoke-PSDepend -Path C:\temp

After a couple seconds, our dependencies are in place:

Invoke-PSDepend C:\temp\requirements.psd1 -Test –Quiet

True

We can verify that the Target for PSDeploy was added to the PSModulePath:

$ENV:PSModulePath

C:\ProjectX;C:\Users\wframe...

Perhaps we want to import these dependencies, and validate that they imported:

Invoke-PSDepend C:\temp\requirements.psd1 -Import

Get-Module -Name BuildHelpers, PSDeploy |
    Select-Object -Property Version, Path

Version Path
------- ----
0.0.20  C:\Program Files\WindowsPowerShell\Modules\BuildHelpers\0.0.20\BuildHelpers.psm1
0.1.21  C:\ProjectX\PSDeploy\PSDeploy.psm1

Enough blather, let’s cover a practical scenario where this might come in handy!

Example: Module Based Virtual Environments

PSDepend can help create minimalistic virtual environments in a few ways:

Prepending $ENV:Path with one or more paths
Prepending $ENV:PSModulePath with one or more paths
Importing modules
Installing modules and files in deterministic locations

Let’s pretend we’re working on ProjectX. Our systems may have different versions of modules, perhaps due to conflicting requirements from other projects. This is a common theme with languages like Python and Ruby, and will likely become more of an issue in PowerShell.

Do we just replace those modules and break things? Do we need to include our own custom code for dependencies with each of our projects, perhaps using PowerShell 5’s side-by-side versioning? No thank you, I’ll try this with PSDepend!

Demo

Once we’ve identified our dependencies, we build a requirements.psd1 file:

@{
    # Set some global, override-able defaults
    PSDependOptions = @{
        Target = 'C:\ProjectX'
    }

    # Grab some modules
    PSSlack = '0.0.15'
    ImportExcel = '2.2.7'
    'Posh-SSH' = 'latest'

    # Download a GitHub repo
    'ramblingcookiemonster/PowerShell' = 'master'

    # Grab an internal module
    'PSAMS' = @{
        DependencyType = 'FileSystem'
        Source = '\\FileServer01\PowerShell\PSAMS'
    }

    # Download a file
    'psrabbitmq.dll' = @{
        DependencyType = 'FileDownload'
        Source = 'https://github.com/RamblingCookieMonster/PSRabbitMq/raw/master/PSRabbitMq/lib/RabbitMQ.Client.dll'
    }
}

I’ll save this in C:\ProjectX\requirements.psd1, and invoke it:

Invoke-PSDepend C:\ProjectX -Force

We can verify that we have the resulting files:

Invoke-PSDepend C:\ProjectX -Test –Quiet

True

Get-ChildItem C:\ProjectX | Select-Object -Property Name

Name
----
ImportExcel
Posh-SSH
PowerShell
psams
PSSlack
RabbitMQ.Client.dll
requirements.psd1

Finally, if we need these modules in our project, we can import them with a one liner:

# Import dependencies!
Invoke-PSDepend C:\ProjectX –Import -Force

# Trust, but verify
Get-Module -Name PSSlack, ImportExcel, Posh-SSH, PSAMS

ModuleType Version Name        ExportedCommands
---------- ------- ----        ----------------
Script     2.2.7   ImportExcel {Add-WorkSheet, BarChart, Col...
Manifest   1.7.6   Posh-SSH    {Get-SCPFile, Get-SCPFolder, ...
Script     0.0.1   psams       {Get-AmsLab, Get-AmsLabMember...
Script     0.0.15  PSSlack     {Find-SlackMessage, Get-PSSla...

That’s it! A small requirements.psd1 file, and we now have our modules loaded and available to re-use in C:\ProjectX, even if our $ENV:PSModulePath includes these modules at incompatible versions.

Next Steps

So! An early version of PSDepend is out there, you can find the raw code on GitHub.

If this is something you might use, kick the tires a bit! Feel free to submit an issue for an idea or bug, or a pull request to fix bugs, code smells, documentation, lack of tests, or anything else that you think might help.

Cheers!

Thumbnail credit

PSDepend: PowerShell Dependencies was originally published by Warren F at Rambling Cookie Monster on January 06, 2017.

Quick Hit: Extract Scheduled Task Credentials

2016-09-09T07:00:00+00:00

Overview

Rambling
Get-Secret
Scheduled Task Credentials
- Wrapping up

Rambling

I enjoy writing. I tend to try to write more than a handful of paragraphs, and often pair things up with a module or guide of sorts. That takes a bit longer than writing a quick hit on something fun I’ve learned.

This Quick Hits section will include bits longer than a tweet, and shorter than a typical post.

Quantity over quality, right? : )

Get-Secret

So! At $LastJob, I managed to convince $Boss that password management was somewhat important.

Among other motivators, I think these stuck out:

Authentication is a thing. With KeePass or a similar solution, anyone who knows the master password gets in, and can take the safe when they leave
Authorization is a thing. With KeePass or a similar solution, you’re stuck with multiple archives if you want to share different passwords with different folks. Which password do you need? Every one!
APIs are a thing. Okay. KeePass and similar solutions might be able to limp by on this, but an authenticated web API designed for multiple users and the works is a bit nicer. Secret Server and PasswordState already have community PowerShell modules wrapping their APIs. Sort of important for this last bullet:
How fast can you change your passwords? John left. He’s angry and has always been a bit rash. He has all the passwords, because you used KeePass, and you don’t know which passwords he’s accessed before he left. Or, maybe Jane hacked through your defenses and made off with your passwords. Whatever the case: Can you cycle all your passwords? How quickly? Will you cause an outage? A real password management solution can help with this, by providing a robust API, perhaps natively hooking into certain technologies for automated credential changes, providing audit logs that tell you which passwords John accessed to limit the scope of these changes, etc.

Anyhow! That was completely tangential, but I miss having a real password management solution, and the automation this enables.

That brings us to todays topic: Shoot. What’s that account’s password again? It’s running a scheduled task, surely we can find it!

Scheduled Task Credentials

Sometimes it can come in handy to extract a password from a scheduled task. Perhaps you forgot to record or update it, because you’re not automating these things yet. Perhaps someone changed it manually and your monitoring hasn’t picked up the mismatch between your password solution’s idea of the password, and the actual password.

I ran into one of these cases, and needed to find an account’s password, without changing it. I’m not going to go into the technical details on how or why this works, I’ll leave that to you. Here’s a quick walk through on extracting a password from a scheduled task:

Have privileges on the system
Download psexec
Download nirsoft’s netpass
Use psexec to launch netpass in SYSTEM’s context

# psexec is somewhere in my $ENV:Path
# netpass is in the root of C:\
# adjust as needed
psexec -i -s -d C:\netpass.exe

Voila! I have the password for this fake scheduled task.

Wrapping up

Tools and references abound - Do consider consulting your $Boss before taking this route. Just in case.

Cheers!

Quick Hit: Extract Scheduled Task Credentials was originally published by Warren F at Rambling Cookie Monster on September 09, 2016.

Invoke PowerShell on Azure VMs

2016-07-21T07:00:00+00:00

Overview

Rambling
Clones
Public IPs
Catastrophe(ish)
Invoke a Script

Rambling

I cut my teeth in a conservative enterprise environment. Few, if any services were hosted. I’m now in an organization that’s very open to hosted services where they make sense.

Need to spin up a bunch of temporary compute? Use something like Azure or AWS, rather than going through the time and effort to provision physical resources that won’t necessarily have a purpose in a few weeks, assuming our processes for physical equipment are even efficient enough to get these up and running in time for the customer’s deadline.

So! I had my first go at this the other week. There’s a deadline coming up, and a team is a bit behind on generating some data. We have plenty of *nix compute, but they have this wonderful single threaded, Windows-specific workload to run.

If only we had a service where we could spin up a bunch of VMs. Oh! We have an Azure account. If only we had the budget. Oh! We might have credits, or maybe this actually made the budget.

If only we had experience spinning up legacy Windows systems in the cloud, with no connectivity to our configuration management platform. Time to improvise and make a few mistakes!

This is a quick hit on my experience with a similar scenario, that will likely expose my inexperience with Azure, and hosted services in general.

Clones

First things first! We spin up a single Azure VM using the legacy Server 2008 R2 OS. Requirements dictate this, sadly. Testing goes well. We’ll take 50 of them! our customer enthusiastically exclaims. After validating that yes, they actually do need 50, selecting an optimal VM size, and configuring the image, we’re off to the races.

Azure quickstart templates look fantastic, and I’ve heard great things, but time is tight, so I borrow some code from Stephane Lapointe to capture a template and create VMs (similar content from Microsoft: capture, create).

There are a few tweaks to make: we use network security groups, and we already have a virtual network to use, but by and large our code is similar, outside of some sanitation and error handling to avoid a mess. We spin up the first clone.

Oh. That VM took over five minutes to create. I’m new to this, so I want to watch things progress to see if any issues come up. I’m not waiting 50 * 5 minutes. Runspaces are a thing. Usually I would go with Boe Prox’s fantastic PoshRsJob, but this is a quick hit, so I go with Invoke-Parallel. Someone mentioned that they had seen it used with Azure, so I wrapped my code in Invoke-Parallel, and kicked things off.

Nice! Fifty VMs up and running. But… how do I get to them?

Public IPs

This has to be simple. There are so many AzureRm Cmdlets, We must be able to pull details on VMs along with their public IPs.

Get-AzureRmVm @params | Get-AzureRmPublicIP @otherparams
# Nope

Get-AzureRmVm @params | Get-AzureRmNetworkInterface @otherparams
# Nope

Huh. There’s no PowerShell-y way to do this that I can find. That’s sad. Guess we’ll write a function! Get-AzureRmVmPublicIp

Get-AzureRmVmPublicIP -ResourceGroupName 'my-resource-group'

# VMName  NICName    PublicIP
# ------  -------    --------
# VM-2    VM-2-NIC   23.96.1.2
# VM-3    VM-3-NIC   23.96.1.3
# VM-4    VM-4-NIC   168.61.2.1
# VM-16   VM-16-NIC  168.61.10.27
# VM-17   VM-17-NIC  23.96.17.56
# VM-18   VM-18-NIC  23.96.19.71
# VM-1    VM-1-NIC   Not Assigned

Perfect! We can now get a list of VMs and public IPs. Our customer doesn’t have access to Azure, that would likely be a better solution here, but what can you do.

Catastrophe(ish)

Our customer is quite thankful. Their work is chugging along on day one. We continue our various other projects and tasks. Ideally we might have something like OMS up and running to watch these VMs. Thankfully, one of our astute customers notices when one of these VMs restart overnight.

We should probably mention here that when you run a workload in the cloud, that workload should be designed appropriately. If you have a system sensitive to restarts running on Azure, AWS, or any other hosted service, you’re going to have a bad time. Alas, we’re talking code written by and for a unique audience; it can’t handle multiple cores, let alone the many scenarios that come up in a hosted environment.

What happened? The sysprep process re-enabled automatic updates. Lesson: If you don’t use configuration management, you need to think carefully about all of the things you previously configured and take for granted.

No problem! We’ll just use PowerShell remoting to fix the config, and to find systems that restarted to give the customer a nice report on systems they’ll need to visit. Oh. Server 2008 R2. Remoting isn’t enabled out of the box. And our group policy isn’t applied.

So! A few lessons. Planning is important. Had we done this before, and not been under a deadline of a day or so, hopefully we would have addressed these. Being able to manage your systems is somewhat important.

That’s fine, Microsoft learned from VMware’s Invoke-VMScript (presumably) and recently gave us PowerShell Direct. Surely this, or something similar is available in Azure.

Nope nope nope.

Invoke a Script

That’s all I want. To invoke a script on a VM. I look around. I read about custom script extensions. Interesting: upload a file, add the extension to a VM, not terribly complicated. Oh. You can only have one per VM. Any other options?

I ask around. I join an Azure Slack team. Crickets. Someone finally responds: Azure Runbooks or Hybrid Runbook Workers might work. I could be wrong, but I couldn’t find anything that would let me run a PowerShell script on an Azure VM, without configuring remoting or registering a hybrid runbook worker on each VM.

Back to custom script extensions. We can deal with the limitations. A few minutes later, updates are disabled. The customer already ran through and checked for restarted systems, so they’re good to go, no list needed.

I have this nagging feeling. I need to be able to run a command and get the output back. We’re not about to RDP into 50 systems manually if something else comes up. We already have the building blocks from executing the CustomScriptExtension to configure updates, all we need to do is abstract out each step of the process.

I check the Azure Slack team one more time. I’m a fan of using existing libraries, and contributing to them if they need a bit more functionality. No luck.

I spend a few minutes and put together Invoke-AzureRmVmScript. I test it on one VM. It works. I test it with Invoke-Parallel. It works. Yay!

Invoke-AzureRmVmScript

We boil everything down into a few high level steps:

This is a quick-publish, so more work to do, but if you’re looking for a simple way to invoke PowerShell on an Azure VM, without using PowerShell remoting or installing a worker on the VM, Invoke-AzureRmVmScript might do the trick.

Let’s look at a few examples!

Investigating Output

PowerShell lets us send output to various streams. I wonder which we can get back from Azure?

    $params = @{
        ResourceGroupName = 'My-Resource-Group'
        VMName = 'VM-22'
        StorageAccountName = 'storageaccountname'
    }

    Invoke-AzureRmVmScript @params -ScriptBlock {
        "Hello world! Running on $(hostname)"
        Write-Error "This is an error"
        Write-Warning "This is a warning"
        Write-Verbose "This is verbose!" -Verbose
        Write-Host "This is killing a kitten"
    }

I wait patiently (this is not fast, Set-AzureRmVmCustomScriptExtension takes some time):

ResourceGroupName : My-Resource-Group
VMName            : VM-22
Substatuses       : {Microsoft.Azure.Management.Compute.Models.InstanceViewStatus, Microsoft.Azure.Management.Compute.Models.InstanceViewStatus}
StdOut_succeeded  : Hello world! Running on HF-22\nWARNING: This is a warning\nVERBOSE: This is verbose!\nThis is killing a kitten
StdErr_succeeded  : C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Downloads\0\017\n21f1b_7a06_4a45_8d12_3974b59deaf5.ps1 : This is an error\n    + CategoryInfo 
                             : NotSpecified: (:) [Write-Error], WriteErrorExcep \n   tion\n    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio \n   
                    n,01721f1b_7a06_4a45_8d12_3974b59deaf5.ps1\n 

Yuck! Unfortunately, we’re getting plain text back here, including special characters like \n which would typically be represented by an actual new line. We might write something to serialize output to JSON if we were so inclined, and were stuck on PowerShell 2 without ConvertTo-Json.

What did we get? Pretty much everything went to StdOut - Output, Warning, Verbose, and Information streams were all captured. As expected, the error stream went to StdErr.

So! That’s one VM, and it took a bit over a minute to run. That’s painful.

Parallelize!

I’m impatient when it comes to running code. Waiting for AppVeyor to queue, test, and deploy a module occasionally induces twitching.

In this case, there’s a simple solution: runspaces! Not writing our own, that can be painful, but borrowing something like PoshRsJob. We’ll use Invoke-Parallel for this simple example:

# Get all the VMs in our resource group where the agent status is ready
$ResourceGroupName = 'My-Resource-Group'
$StorageAccountName = 'mystorageaccount'
$StorageAccountKey = 'my storage account key in plain text' # you can omit this and we'll pull it for you...
$VMs = Get-AzureRmVM -ResourceGroupName $ResourceGroupName |
    Foreach {
        Get-AzureRmVm -ResourceGroupName $ResourceGroupName -Name $_.Name -Status
    } |
    Where-Object {$_.VMAgent.Statuses[0].DisplayStatus -like 'Ready'} |
    Select -ExpandProperty Name

Basically, I want all VMs in my resource group that have a VMAgent that is ready. I have my VMs, time to play!

# Kick off script on VMs in parallel
# 50 at a time, timeout at 5 minutes, pull in variables

$InvokeParallelParams = @{
    RunspaceTimeout = (60*5)
    Throttle = 50
    InputObject = $VMs
    ImportVariables = $true
}

$Output = Invoke-Parallel @InvokeParallelParams -ScriptBlock {

    # Load Invoke-AzureRmVmScript. Alternatively you could hard code it here, or we could fix Invoke-Parallel to pull in functions
    . 'C:\Invoke-AzureRmVmScript.ps1'

    # Parameters to splat
    $params = @{
        ResourceGroupName = $ResourceGroupName
        VMName = $_
        StorageAccountName = $StorageAccountName
        StorageAccountKey = $StorageAccountKey
        Force = $True
    }

    Invoke-AzureRmVmScript @params -ScriptBlock {
        $Processes = Get-Process -Name ExampleProcess
        "'$($Processes.count)' ExampleProcess processes on $(hostname)"
    }
}

It will still take some time, but this beats running things serially! It looks like at least one VM is ready to tear down:

$Output

VMName StdOut_succeeded
------ ----------------
VM-12  '' ExampleProcess processes on VM-12
VM-21  '4' ExampleProcess processes on VM-21
VM-18  '5' ExampleProcess processes on VM-18
VM-40  '5' ExampleProcess processes on VM-40
VM-27  '6' ExampleProcess processes on VM-27
VM-3   '6' ExampleProcess processes on VM-3
VM-51  '6' ExampleProcess processes on VM-51
VM-52  '6' ExampleProcess processes on VM-52
VM-6   '6' ExampleProcess processes on VM-6
...

That’s about it!

Going forward, we’ll have more time to plan these out, and connectivity should be available for important things like configuration management and authentication, but if we ever need it, we’ll have a quick tool to hit poorly-setup-systems. Perhaps we could bootstrap remoting or other connectivity with this function, but these VMs are already reaching their end of life.

And thus ends my first experience working with resources in the cloud. It’s been fun, looking forward to more!

Side note: the brown cloud illustrates my ability to work with the cloud, and is not a reflection on Azure. Okay, maybe it’s a hyperbolic reflection on the documentation, a few bugs I ran into, and some not-as-PowerShell-y-as-expected behavior.

Invoke PowerShell on Azure VMs was originally published by Warren F at Rambling Cookie Monster on July 21, 2016.

A PowerShell Module Release Pipeline

2016-07-14T12:00:00+00:00

Overview

Rambling
Why Continuous Deployment for Modules?
The Ingredients
The Recipe
Following the Recipe
What Happens Now?
Rambling Outro: The Gallery

Rambling

I like this new Microsoft. Not only are they open sourcing more components, taking pull requests, and contributing upstream, they’re doing a great job reaching out and interacting with the community.

In the past year or so, I went from occasionally throwing code on GitHub, to collaborating on open source projects, sharing what I learned, and somehow having a fun project of mine appear in a few sessions on release pipelines.

I owe much of this to Sergei V. and Michael G., two Microsoftees who helped motivate and prod me (and others) along the way.

This post is a follow up to the series on GitHub, Pester, and AppVeyor, and the post on building PowerShell modules. We’re going to wrap up the module writing process and demonstrate how to automatically deploy your modules to the PowerShell Gallery.

Why Continuous Deployment for Modules?

The importance of writing modules is covered here. This misses a few pieces of the puzzle though: now that we have a module up on GitHub and published to the gallery, we’re not quite done. If we want to reap some of the benefits of open source, we need to make it easy to pull in suggestions from helpful folks on the Internet.

For those of you writing modules, how often have you taken a pull request or committed a change, thinking I’ll get around to publishing this on the PowerShell gallery later? Publishing to the gallery is fairly painless, but it’s another thing to do and to remember. We’re all human, so we occasionally forget, or get lazy and skip that step.

You might even add those extra steps to your calculation of how long it will take to integrate someone’s suggestions. I’ll have to merge that in, and do a bunch of things afterwards to publish it, I’ll just tackle it all some other time.

Having a pipeline to automatically test and deploy our changes can remove the barriers that might dissuade us from these changes. It’s also a great learning opportunity for CI/CD ideas that you could apply to your infrastructure.

So! Where do we start?

The Ingredients

Let’s look at the ingredients for a recipe that gives us continuous deployment for a PowerShell module. We’ll use content from PSDeploy itself as an example:

A module built roughly following this guide
A GitHub account
An AppVeyor account (sign in with GitHub)
A PowerShell gallery account
Your PowerShell gallery key, encrypted by AppVeyor
appveyor.yml. This tells AppVeyor what to run when you make a change to your module
Build.ps1. This is a simple script to pull in dependencies and kick off psake, which does the real work
psake.ps1. This organizes your build into “tasks”. It runs your tests and deployments
deploy.psdeploy.ps1. This tells PSDeploy how to deploy your project - in this case, publishing a module

Let’s put these together!

The Recipe

Let’s boil this down into a few steps:

Create your module
Sign up for AppVeyor and PowerShellGallery.com
Add scaffolding for a release pipeline
Profit

That’s it! Once you get your module set up from the previous article, you can just layer a little scaffolding on top to automatically publish to the PowerShell gallery.

Following the Recipe

AppVeyor and PowerShell Gallery

So! You should already have a GitHub account. We need two more accounts:

Register at PowerShellGallery.com. Sign in
Copy your PowerShell Gallery API key from your account page. Keep this key a secret
Sign in to AppVeyor with your GitHub account
Create a secure variable: Click your AppVeyor account drop down, Encrypt data. Paste in your API key and Encrypt! Copy out the resulting encrypted value

Note that GitLab CI and other build systems often have ways to inject secure data like this.

That’s it for the GUI stuff, it’s time to dive into some PowerShell!

Release Pipeline Scaffolding

We’re going to use the following components for our release pipeline:

AppVeyor.yml. Instructions for AppVeyor. We’ll still use this, but we’ll try to move as much of the build as possible into PowerShell tooling that will work in other build systems.
Build.ps1. A build script that sets up our dependencies and kicks off psake. Portable across build systems. We install and use a few dependencies:
- BuildHelpers. A module to help with portability and some common build needs
- Psake. A build automation tool. Lets us define a series of tasks for our build
- Pester. A testing framework for PowerShell
- PSDeploy. A module to simplify PowerShell based deployments - Modules, in this case
Psake.ps1. Tasks to run - testing, build (e.g. bump version number), and deployment to the PowerShell gallery
deploy.psdeploy.ps1. Instructions that tell PSDeploy how to deploy our module

This combination meets two goals:

It’s generalized and can be dropped into a new project
Outside of adding a build-system-config that kicks off build.ps1, it can be used on build systems like Jenkins, GitLab CI, or even your own PC

This might seem complex, but it’s really just a few steps - let’s walk through the code:

AppVeyor.yml

There are two key bits in the yaml:

environment:
  NuGetApiKey:
    secure: oqMFzG8F65K5l572V7VzlZIWU7xnSYDLtSXECJAAURrXe8M2+BAp9vHLT+1h1lR0

This decrypts our API key and creates an environmental value that we can call in PowerShell as $ENV:NuGetApiKey. Be careful not to display this in any output.

test_script:
  - ps: . .\build.ps1

This runs our build.ps1 file. AppVeyor runs from the project root, so we know the relative path to this script.

Build.ps1

In build.ps1, we use Resolve-Module from Brandon Padgett to pull in dependencies: BuildHelpers, psake, Pester, and PSDeploy:

# Grab nuget bits, install modules, set build variables, start build.
Get-PackageProvider -Name NuGet -ForceBootstrap | Out-Null

Resolve-Module Psake, PSDeploy, Pester, BuildHelpers

Once these are in place, we use Set-BuildEnvironment to create some environment variables for our project. Here’s some example output:

Name                 Value
----                 -----
BHProjectName        psdeploy
BHProjectPath        C:\projects\psdeploy
BHPSModuleManifest   C:\projects\psdeploy\psdeploy\psdeploy.psd1
BHPSModulePath       C:\projects\psdeploy\psdeploy
BHCommitMessage      !Deploy Brandon's changes
BHBuildSystem        AppVeyor
BHBranchName         master
BHBuildNumber        132

You can use these details to simplify the rest of your build code, and to normalize variables across build systems. At the moment BuildHelpers supports AppVeyor, Jenkins, GitLab CI, and VSTS (Thanks Stijn!); pull requests would be welcome.

Lastly, we kick of psake, and if it fails (e.g. a failed test), we exit with a non-zero code to tell our build system that we didn’t succeed.

Invoke-psake .\psake.ps1
exit ( [int]( -not $psake.build_success ) )

Psake.ps1

This is where the real work starts to happen.

First things first, we set up a few variables that we use later - we use $PSScriptRoot if BuildHelpers hasn’t run, set a verbose flag if we see !verbose in a commit message, and run an Init task that doesn’t do much.

Test Phase

Next up we start running some tests:

$TestResults = Invoke-Pester -Path $ProjectRoot\Tests -PassThru -OutputFormat NUnitXml -OutputFile "$ProjectRoot\$TestFile"

If you want a re-usable header for your *.tests.ps1, you can use the output from BuildHelpers to generalize things:

# [Content from *.tests.ps1, not psake.ps1] #
$PSVersion = $PSVersionTable.PSVersion.Major
$ModuleName = $ENV:BHProjectName
$ModulePath = Join-Path $ENV:BHProjectPath $ModuleName

# Verbose output for non-master builds on appveyor
# Handy for troubleshooting.
# Splat @Verbose against commands as needed
$Verbose = @{}
if($ENV:BHBranchName -notlike "master" -or $env:BHCommitMessage -match "!verbose")
{
    $Verbose.add("Verbose",$True)
}

Import-Module $ModulePath -Force

Back in psake.ps1, we check to see if $ENV:BHBuildSystem is AppVeyor, and push up test results if so. This isn’t critical, but it gives us a list of test results on AppVeyor.

Build Phase

After the tests run, we can start a build phase. Not everyone will want to use this, but I like to run two shortcuts:

Bump the module version
Update FunctionsToExport = '*' in the module manifest to include all exported functions

# Load the module, read the exported functions, update the psd1 FunctionsToExport to include exported functions
Set-ModuleFunctions

# Bump the module version
Update-Metadata -Path $env:BHPSModuleManifest

Deploy Phase

Now for the fun part! We deploy the module:

$Params = @{
    Path = $ProjectRoot
    Force = $true
    Recurse = $false # We keep psdeploy.ps1 test artifacts, avoid deploying those : )
}
Invoke-PSDeploy @Verbose @Params

That was a bit anti-climactic! We’ve offloaded all the logic for publishing our module to PSDeploy, leaving us cleaner code.

Deploy.PSDeploy.ps1

We have two deployments in here. If we’re in a recognized build system ($env:BHBuildSystem), in the master branch ($env:BHBranchName), and have a commit message that includes !deploy ($env:BHCommitMessage), we publish our module to the gallery:

Deploy Module {
    By PSGalleryModule {
        FromSource $ENV:BHPSModulePath
        To PSGallery
        WithOptions @{
            ApiKey = $ENV:NugetApiKey
        }
    }
}

This is great, but wouldn’t it be nice to have development builds that folks could test and try out? We can borrow from the PowerShell team’s idea (and code!) of deploying NuGet packages to AppVeyor and add a second deployment - sounds complicated and fancy, but it’s not too bad:

Deploy DeveloperBuild {
    By AppVeyorModule {
        FromSource $ENV:BHPSModulePath
        To AppVeyor
        WithOptions @{
            Version = $env:APPVEYOR_BUILD_VERSION
        }
    }
}

Hit the docs for details on how to install a module published to AppVeyor.

That’s it! We can now install our official module from the PowerShell Gallery, and install development builds from AppVeyor.

What Happens Now?

This whole process sounds complicated, but it’s driven by four generic files you can add or update in your repository: appveyor.yml, build.ps1, psake.ps1, and something.psdeploy.ps1.

Here’s what happens automatically with every commit we push to GitHub:

GitHub sends AppVeyor a notification of your commit
AppVeyor parses your appveyor.yml and starts a build on a fresh VM
build.ps1 installs dependencies, sets up environment variables with BuildHelpers, and kicks off psake.ps1
psake.ps1 does the real work. It runs your Pester tests, and if they pass, runs PSDeploy against your psdeploy.ps1

That’s about it! Once this scaffolding is in place, you can let GitHub and AppVeyor do the work for you, and start thinking about applying release pipelines like this to your infrastructure!

Rambling Outro: The Gallery

Today, the PowerShell Gallery has fewer than 1,000 modules, many of them focusing on Azure and DSC. Hardly comparable to repositories like CPAN, PyPI, or RubyGems:

Hopefully we can get to a point where more folks are contributing to the gallery. There are a few things that might move this along:

Using tools like this pipeline could simplify and automate publication, encouraging lazy folks like myself to use the gallery
Educating our peers and outdated legal teams on the importance of open source could unlock a trove of internal tools that would be quite valuable to the community. Chef offers a webinar for lawyers; more education targeted at this common road-block-to-sharing might be helpful
Microsoft could lead by example. Do you see OperationsManager, SqlServer, SqlPs, or Microsoft.SharePoint.Powershell on the gallery?
You could pester your vendors to distribute modules on the gallery. More should be writing modules, and those that do rarely publish them on the gallery. Props to teams like Amazon’s who already do this
Open sourcing PowerShell could open the doors to pragmatic OSS contributors who have no motivation to work with an OS-specific language

Cheers!

A PowerShell Module Release Pipeline was originally published by Warren F at Rambling Cookie Monster on July 14, 2016.

PowerShell and Slack

2016-08-30T11:00:00+00:00

Overview

Rambling
- Why Chat?
- Slack
PSSlack
- PSSlack Prerequisites
- PSSlack Installation
Using PSSlack
Next Steps

Rambling

I grew up using IRC. Before Napster and Steam, there were other ways to find music and games. As time went by, I shifted over to more legitimate use, with freenode and other networks offering solid technical resources, including channels for PowerShell, VMware, Citrix, and more.

At some point, the Slack craze started, and Joel Bennett created the PowerShell Slack team (invite link). It took some convincing, but sometimes I just want a pretty, simple to use solution that works across platforms. Slack fit the bill, so I hopped on!

Eventually, I ended up on a team that uses Slack for communication. Perfect excuse to write a Slack module!

Why Chat?

So! Let’s step back a moment and consider why you might want a synchronous group chat system… Let’s use two example cases. Which would you prefer:

Case 1:

A critical issue occurs. You’re heading to a physical war room. Not a loud person? Good luck getting your voice in there. Love repeating the status? Perfect, you’ll do this each time someone new steps in the room. Want a comfy keyboard and monitor? Ha, enjoy your laptop. Good luck writing your postmortem from this havoc
Something big happens, but doesn’t warrant a war room. Get ready for 10 different e-mail threads with different recipients and miscommunication galore, and a ticket or two with duplicate but not complete information. Good luck piecing things together!
You want to work remotely. You’re stuck with e-mail, phone calls, and 1:1 oriented IM with something like Lync. This isn’t condusive to communication
You check a variety of sources. You have a logging solution, a monitoring solution or three, ad hoc e-mail notifications from scripts that go straight to junk mail, and more!

Case 2:

A critical issues occurs. You head to a generic emergency chat room, or maybe create a new one specifically for this issue. You don’t need to interrupt people to raise an important point. Anyone who joins in can scroll up and read the full transcript. You can work from the comfort of wherever you want to work. When all is said and done, you have a good starting point for a postmortem
Something big happens, but doesn’t warrant a war room. You hop on chat, maybe join a generic war room, and hash things out. You can mention or invite folks to join, and everyone can see the transcript, rather than try to parse 10 e-mail threads. You can add the chat transcript to your ticketing system, if that’s your thing
You want to work remotely. If everyone is using chat, you can keep up to date with what everyone is up to, start up discussions, and generally have an easier time communicating with your team, regardless of where you are
You check a variety of sources. But many or most of them funnel into a channel or two that you can view to get a quick read on things. You can search these things after the fact
Your chat system is likely hosted. This means when shit hits the fan, you have a medium that isn’t depending on your infrastructure to run (on-prem IRC? No thanks)

Oh. And if you go beyond sending messages to your chat system, you can do some crazy fun things with a bot. Check out this awesome bit from Matt Hodgkins on setting up Hubot with PowerShell

So! Hyperbole (or is it?) aside, consider a group chat system; they can be incredibly helpful. There are a variety of options out there. Slack and HipChat are two of the better solutions out there.

Slack

Slack is a solid ~~gif library~~ chat solution. It has a slick client for most platforms, and a huge library of integrations. That being said, you might run into cases where you just want to send a message to Slack. Why?

Maybe the system you want to hook into Slack doesn’t have an integration
Maybe the existing integration is unhelpful, and blends into a sea of other messages that looks the same
Maybe the existing integration doesn’t offer enough customization
Some mix. SCOM is a great example. No Slack integration, and even if there were, you could add context to SCOM alerts and send them along to Slack, perhaps in addition to pretty HTML e-mails

Wait. But why would I send my stuff to Slack? It turns out that this is quite helpful.

Who wouldn’t want to have informational logs, various levels of alerts, conversation, and other data, all in one spot, search-able from any client, without worrying about it hitting you junk mail, being archived, or being spread across your various logging, monitoring, mail, file systems, event logs, and other sources that usually house this data? Sure, use those other solutions, but do consider how helpful it is to send this stuff to a chat system.

Isn’t this a PowerShell post, where’s the PowerShell?

PSSlack

Like any project, I like to look around to see what’s available in the community. There might be an existing module, or at least some functions that you could contribute to or borrow from.

I found an awesome function or two from jgigler, and a handy module from Steven Murawski, but I was looking for a bit more functionality.

I borrowed some work and ideas from jgigler and Steven, and put together a quick and dirty PSSlack module! Let’s do a quick walk through.

PSSlack Prerequisites

First things first, we need a way to authorize with Slack.

You can read more, but we’re just going to grab a test token for this exercise.

Make sure you’ve signed into your Slack team in the browser (not just the app)
Browse to this page, create a token, and copy it! Don’t give this out. Treat it like your username+password

If you’re just sending messages, you could also set up an incoming webhook and use the Uri (which you should also treat like a username+password).

PSSlack Installation

The next part is easy! We’ll use the PowerShell Gallery to grab the module (works with Windows 10, WMF 5, or a separate MSI installer at the link). You could also pull it down from GitHub.

Install-Module PSSlack -Force

That’s it! Now you can start exploring the commands and help:

# Import the module.
    Import-Module PSSlack    #Alternatively, Import-Module \\Path\To\PSSlack

# Get commands in the module
    Get-Command -Module PSSlack

# Get help
    Get-Help Send-SlackMessage -Full
    Get-Help about_PSSlack

Using PSSlack

We’re done with the boring bits, lets look at the actual module! First things first, you can send a simple message with no bells or whistles:

$token = 'xoxp-some-token'

Send-SlackMessage -Token $token `
                  -Channel '@wframe' `
                  -Parse full `
                  -Text 'Hello @wframe, join me in #devnull!'

Let’s look at some more features.

Rich Attachments

New-SlackMessageAttachment -Color $([System.Drawing.Color]::red) `
                           -Title 'The System Is Down' `
                           -TitleLink https://www.youtube.com/watch?v=TmpRs7xN06Q `
                           -Text 'Please Do The Needful' `
                           -Pretext 'Everything is broken' `
                           -AuthorName 'SCOM Bot' `
                           -AuthorIcon 'http://ramblingcookiemonster.github.io/images/tools/wrench.png' `
                           -Fallback 'Your client is bad' |
    New-SlackMessage -Channel 'devnull' `
                     -IconEmoji :bomb: |
    Send-SlackMessage -Token $Token

In this example, we create a Slack attachment, a way to provide a richer message. We pipe this along to New-SlackMessage, which constructs a Slack message. Finally, we send the newly create message with our existing token.

Multiple Attachments and Fields

Let’s throw in a kitchen sink. We can pipe multiple attachments together, and include a table of fields from any PowerShell object. We made up a fake example here:

#You could pipe anything into New-SlackField.  We'll pretend to pull monitoring alert data:
$Fields = [pscustomobject]@{
    ImpactedService = 'All The Things'
    Severity = 11
    ImpactedDepartment = 'All'
    URL = 'https://www.youtube.com/watch?v=TmpRs7xN06Q'
} | New-SlackField -Short

# Notice that we can chain multiple attachments:
New-SlackMessageAttachment -Color $([System.Drawing.Color]::red) `
                           -Title 'The System Is Down' `
                           -TitleLink https://www.youtube.com/watch?v=TmpRs7xN06Q `
                           -Text 'Please Do The Needful' `
                           -Pretext 'Everything is broken' `
                           -AuthorName 'SCOM Bot' `
                           -AuthorIcon 'http://ramblingcookiemonster.github.io/images/tools/wrench.png' `
                           -Fallback 'Your client is bad' |
    New-SlackMessageAttachment -Color $([System.Drawing.Color]::Orange) `
                               -Fields $Fields `
                               -Fallback 'Your client is bad' |
    New-SlackMessage -Channel 'devnull' `
                     -IconEmoji :bomb: `
                     -AsUser `
                     -Username 'SCOM Bot' |
    Send-SlackMessage -Token $Token

That’s about it for messages! Get creative. A while back we replaced our default, contextless SCOM notifications with a nice HTML e-mail notification. It would be simple to hook the same alerts into Slack, for example.

What else can we do?

Searching Slack

Slack has a great built in search, but maybe you want to hook something up that only sends messages if a UID or search term doesn’t appear in Slack already.

We’ll look for PowerShell discussion:

Find-SlackMessage -Token $Token `
                  -Query 'PowerShell' `
                  -SortBy timestamp

There’s a lot more context if we use Select-Object:

Find-SlackMessage -Token $Token `
                  -Query 'PowerShell' `
                  -SortBy timestamp |
    Select-Object -Property *

We can also do something similar with channel history.

Getting History

There was recently a call for PowerShell / DevOps related topics. We’ll use this as a demo for what you might do with history:

Get-SlackChannel -Token $token -Name *topic-requests |
    Get-SlackHistory -Token $token -Count 1000 |
    Where-Object {$_.Reactions.Count -gt 0} |
    Select @{ l='Thumbs'; e={$_.Reactions[0].count}}, Text |
    Sort Thumbs -Descending

We find the channel in question, look for history, only return messages with reactions, add a thumbs property that contains the count of reactions, and sort:

Posting Files

This is a work in progress, but here’s an example of uploading a snippet:

$Snippet = @'
Send-SlackFile -Token $Token `
               -Channel general `
               -Content $Snippet `
               -FileType powershell `
               -filename example.ps1
'@

Send-SlackFile -Token $Token `
               -Channel general `
               -Content $Snippet `
               -FileType powershell `
               -filename example.ps1

Notice that this sends as me - you would need to register an application and grab a real token if you want this to appear from a bot.

Thanks to Brooks’ pull request, we now support sending files, because your Slack team could use a few more gifs!

That’s about it!

Next Steps

There’s more to come. Slack has a nice, well documented API, that offers you a GUI for simple testing of each method. We’re slowly working our way through these methods to see what would be valuable, so be sure to stop back, this module will be changing rapidly.

Suggestions, pull requests, and other contributions would be more than welcome. If you have any ideas, check to see if there’s already a request in the issues, and open an issue!

Cheers!

PowerShell and Slack was originally published by Warren F at Rambling Cookie Monster on May 21, 2016.

PSDeploy: Take Two

2016-04-01T09:00:00+00:00

Overview

Rambling
Why Would I Use PSDeploy?
How Do I Install PSDeploy?
How Do I Use PSDeploy
What Can I Deploy?
What Changed in PSDeploy?
Wrapping Up

Rambling

So! I’m a dad now. Everything went smoothly, Hannah arrived mid-March, at 8lbs 9oz. I’m a light sleeper, so I now have a few hours every night to hack away at things in a sleep-deprived state, lending an occasional hand with our trouble-making daughter.

Right before all the fun, I got a fun message from Michael Greene:

wanted to ask where you want to go with that PSDeploy project. could we get it to a DSL along the lines of “Deploy <type> <path> To <Target>” and then from any project just invoke-deployment.

It sounded interesting, but complicated. I was happy with the simple YAML implementation, but gave the change some thought. A DSL would mean deployment configurations could call PowerShell inline with the deployment. It wouldn’t add much complexity for a user, but it would add some serious flexibility.

This post will be a quick and dirty overview of the revamped PSDeploy.

Why Would I Use PSDeploy?

Maybe you’re an IT professional, and you’re starting to use important tools like version control. This can be a pain:

Do you commit to version control and manually copy things out to where they live in the real world?
Do you edit things where they sit in the real world, and then commit that to version control?
Do you ever let these get out of sync?

A simple answer to this is to borrow an idea that developers have been using for a while: continuous deployment.

When you commit to version control, a tool like Jenkins, TeamCity, AppVeyor, Octopus Deploy, or other CI/CD solutions will see this, and can automatically deploy your changes out to production. Ideally after some tests run successfully. PSDeploy can help simplify that deployment.

So! What does this actually look like for an IT professional? Here’s a simple example:

I make a change to a scheduled task script on my computer. I push those changes to version control, and the rest is automatic! In the background, Jenkins sees my change, calls PSDeploy, and the scheduled task script is pushed out to production!

Here’s an illustration from the original PSDeploy post:

How Do I Install PSDeploy?

You can pull down PSDeploy a few ways.

The easy way, assuming you have PowerShell 5, or install the down-level PowerShellGet module:

Install-Module PSDeploy

Import-Module PSDeploy

The old-fashioned way:

Download the repository archive
Unblock the archive file
Extract the PSDeploy folder to a module path (e.g. $env:USERPROFILE\Documents\WindowsPowerShell\Modules\)
Import the module: Import-Module PSDeploy

Once you’ve installed PSDeploy, you can check out the various functions and help:

# Get commands in the module
    Get-Command -Module PSDeploy

# Get help for the module and a command
    Get-Help about_PSDeploy
    Get-Help Invoke-PSDeploy -full

That’s about it, you’re ready to get started with PowerShell based deployments.

How Do I Use PSDeploy

So! You’ve installed PSDeploy. Now what?

At a high level, you write a *.psdeploy.ps1 deployment script. PSDeploy reads this and runs the deployments that you defined.

Let’s step through a few examples to illustrate the basics. We’ll pretend we have the following files and folders for these examples:

A basic deployment

At the very least, we need a Deploy, By, FromSource, and To:

Deploy SomeDeploymentName {
    By FileSystem {
        FromSource MyModule
        To C:\PSDeployTo
    }
}

You can check what PSDeploy sees by running Get-PSDeployment:

Get-PSDeployment -Path C:\PSDeployFrom\my.psdeploy.ps1

Let’s add some more features!

A more fun deployment

We’re going to use a few new features of PSDeploy: tags and dependencies. They’re relatively pointless here, but might help you with an actual deployment need:

Deploy SomeDeploymentName {
    By FileSystem AllTheThings {
        FromSource MyModule,
                   SomeScripts
        To C:\PSDeployTo
        DependingOn SomeDeploymentName-MyModule  #DeploymentName-ByName
        Tagged Dev
    }

    By FileSystem MyModule {
        FromSource MyModule
        To \\ServerY\c$\SomePSModulePath,
           \\ServerX\SomeShare$\Modules
        Tagged Prod,
               Module
    }
}

Let’s look at how PSDeploy sees this:

Like many PowerShell objects, we can find more properties using Select-Object:

So! We see that the deployments have tags, and Get-PSDeployment gave us the results back taking the dependency we specified into account.

Finally, maybe we only want to look at deployments tagged prod:

So, how do we actually run these deployments?

Invoke-PSDeploy

Invoking a deployment is simple: we call Invoke-PSDeploy. This borrows a few conventions from Invoke-Pester.

Running Invoke-PSDeploy with no parameters will recursively search for *.psdeploy.ps1 files under your current path, and run everything that it finds. In this scenario, any relative paths in the deployment use the current path as the deployment root:

Invoke-PSDeploy

Running Invoke-PSDeploy with a path parameter specifying a folder will recursively search for *.psdeploy.ps1 files under that specific folder. It will use the specfied folder as the root for any relative paths in the deployment:

Invoke-PSDeploy -Path \\Path\Below\PSDeployConfigs\

Finally, you can run Invoke-PSDeploy with a path parameter specifying a *.psdeploy.ps1 file. It will use that deployment script’s parent path as the root for any relative paths in the deployment:

Invoke-PSDeploy -Path \\Path\To\Some.PSDeploy.ps1

What Can I Deploy?

Now that you know how to use PSDeploy, what can you deploy? Some of these things might seem simple, but having a layer of abstraction to hide some of your code can be quite helpful:

FileSystem: Copy files or folders. Uses copy-item and robocopy behind the scenes, respectively
FileSystemRemote: Same as FileSystem, but runs in a remote PSSession. Mind the double hop.
MkDocs: Build and deploy an MkDocs site - thanks to Michael Lombardi!

It’s pretty bare bones for now, but contributions would be more than welcome. Extending PSDeploy takes two quick steps: write a script to handle deployments, and a config file that tells PSDeploy what DeploymentType invokes that script.

It might be a fun way to get experience working with GitHub, and perhaps some integration testing, if you’re up for it!

As time goes on, you can run Get-PSDeploymentType to list available deployment types, and Get-PSDeploymentType SomeDeploymentType -ShowHelp to view the help for that deployment type:

That’s about it! Let’s recap.

What Changed in PSDeploy?

*.PSDeploy.ps1 deployment configurations, in addition to yaml
Yaml deployment configurations deprecated. These still work, but development efforts and new features will focus on *.psdeploy.ps1 scripts
Tagged deployments to help filter or categorize deployments
Deployment dependencies to help ensure deployments perform in the necessary order
Noop DeploymentType to help explore or debug the environment when a deployment runs

Wrapping Up

This was a fun little project! I’ve found the functionality quite helpful for PowerShell based deployments, whether they start in Jenkins, GitLab CI, or from the shell.

If you’d like to contribute or collaborate, issues and pull requests would be welcome! If you want to extend PSDeploy with more deployment types, some brief notes are available in the wiki, and I would be more than happy to help!

Cheers!

PSDeploy: Take Two was originally published by Warren F at Rambling Cookie Monster on April 01, 2016.

On Learning, Sharing, and My First Tech Job

2016-02-23T22:00:00+00:00

Overview

Rambling
- FirstTechJob
Learning
- Rambling
- Takeaway
Sharing
- Rambling
- Takeaway
Closing.

Rambling

Over the weekend, a fun thing happened on Twitter. Someone encouraged folks to post about their first tech job.

It brought to mind two of my primary motivations: learning, and sharing.

FirstTechJob

I lucked out. While I’ve always been enthusiastic about computing, when I first applied for a job in IT, my experience consisted of an unrelated undergrad degree (brains!), a co-op in risk and compliance (not very technical), and being a student in a networking and systems administration program at a local university.

As much as you learn in school, we all know that nothing can make up for actual experience in the real world. I still have no idea what I’m doing, but I had even less of a clue back then.

So! I applied for a few jobs. I was assuming a help desk job would be the only option, probably on an odd shift.

I didn’t hear back from most places. Eventually I took a shot in the dark, and applied for a data interfaces job at a local hospital; something I had no business applying for. I chatted with the hiring manager, the place seemed nice, but I was clearly not the best fit. Thankfully, they ended up passing my resume along, and I got a random call for a site support position.

That was it! I started working with Windows in a domain environment for the first time, helping folks with their laptops, desktops, and other client systems. Somehow, I managed to (mostly) steer clear of printers.

Learning

Rambling

Learning is so important in the world of technology. If you don’t enjoy learning, you’re going to hate working in IT. Or you’ll be that guy/girl, the one who happily resorts to clicking next, no matter how much encouragement and support you give them to learn more and help themselves and their team.

I started out as that guy, more out of ignorance than intention. I tried to eek out my co-workers’ years of wisdom. I learned about important technologies at each job along the way. When I moved up to a third level support role, I learned more about servers and networking, and got a superficial introduction to Active Directory, VMware, and at the suggestion of a mentor, VBscript.

That last bit panned out. I moved into a systems engineer role immediately after the exposure to VBscript, probably way before I was ready. So I dove in! VBscript was a pain, but the value of scripting was clear: thankfully PowerShell was already well established!

Rather than picking a narrow field to specialize in, I decided to focus on something that applies across IT: scripting! I started learning PowerShell, and continue to learn more to this day. One of the many benefits to PowerShell is that you can use it to glue together pretty much any technology. I used this to learn more, and to get involved in interesting projects. SQL Server, Infoblox, Isilon, vCenter, Active Directory, SCOM, you name it.

I’m following the same pattern today, and am starting to to learn *nix toolsets and basics, Python, Ruby, Puppet, Salt, and more. The yaks are lining up.

Takeaway

These helped me out. Take this with a grain of salt, but I suspect they may help you as well:

Be curious. Explore. If you don’t enjoy learning, steer clear of IT
It’s not my job is a terrible excuse. Always be open to learning topics outside of your current role
Find and learn force-multipliers that apply across most jobs in IT. Scripting (e.g. PowerShell), data manipulation (e.g. SQL), and others like these are valuable skills
You don’t need to specialize. Knowing a broad range of topics at shallow to intermediate depth, and a few at extensive depth can be quite valuable

Rambling

Learning and sharing go hand in hand. Many of the resources I found most helpful were shared by other IT professionals: code snippets, functions, tools, libraries, ideas, questions and answers, you name it.

So! I tried to contribute back.

I started out by just sharing tips, tricks, and tools with co-workers. Eventually I picked up PowerShell, and started sharing snippets, and modules (think of them like toolboxes) with a bunch of handy functions (tools) oriented around our environment.

I enjoyed sharing with co-workers, so I took it to the next stage, and started a blog. Nothing too exciting, I mostly shared tech news and ideas to start. A first look at PowerShell Web Access in the wild. Details on the awesome new PowerShell 3. A PowerShell Cheat Sheet. A few months after starting, I released my first function (a fork of Shay Levy’s code), Get-NetworkStatistics.

I’ve been sharing ever since. If a function or module is general purpose, doesn’t include business logic, and is easily sanitized, I publish it. I didn’t explicitly ask my boss, although the topic came up a few times. I’d rather ask for forgiveness than permission, but you might have a tighter risk appetite, or work for a more litigious employer.

I find that topic irksome. Employers benefit from more informed IT practitioners. Preventing IT folks from sharing their work holds back both the employer and IT practitioners who could have benefited from the knowledge. It also creates some uncertainty, where the more risk-averse among us avoid sharing, even if they aren’t under contractual obligation to hoard knowledge.

We all benefit from sharing. Rather than having every IT professional write duplicate tools, with no feedback or contributions from others, it makes more sense to share our code and ideas. Less duplication of effort, more eyes on the code or ideas, more feedback from folks taking your ideas or code and using them in new and different ways; this benefits the practice of IT itself. Heck, it might not be the intent, but if you read the LOPSA code of ethics, they mention how important sharing is:

Takeaway

Again, I found these beneficial, but take them with a grain of salt:

Share. You’ll learn more. You might polish your code and follow practices you would otherwise ignore. You’ll probably want to review information and learn more about the topic at hand before you post about it
Share. You might get helpful feedback and end up with a better solution
Share. You’ll help others in IT, who in turn might share their own ideas and code
Share. It looks good to many employers. All else being equal, if I’m looking at two candidates, and only one of them has code and ideas open to the public, they’ll probably get the nod
Don’t rule out sharing off-hand. Consider the risk and reward of sharing your sanitized code and ideas
Provide digestible details. What’s the goal of your code? How do we install it? How do we use it? Readme.md files or short blog posts go a long way. As silly as it sounds, animated gifs are a great way to quickly get across an idea.

Closing.

That’s about it!

I owe a debt of gratitude to my co-workers, and the various PowerShell and other IT communities and contributors out there. If they hadn’t shared their knowledge, I wouldn’t have had the same resources to learn with, and wouldn’t be able to share back the little that I can.

Don’t be afraid of learning something new, and share what you’re learning along the way. I went from “what’s PowerShell?” to “Why PowerShell?.” in a few short years. Years sounds like a long time, but spending a little time every day was all it took.

Cheers! Hopefully back to posting more technical content soon.

On Learning, Sharing, and My First Tech Job was originally published by Warren F at Rambling Cookie Monster on February 23, 2016.

PowerShell Is Too Hard

2016-02-18T22:00:00+00:00

Overview

Rambling
The Task
You Don’t Need to be a Developer to use PowerShell
- How do I get started?

Rambling

I lucked out and no longer need to convince co-workers or their managers that it’s in their best interest to learn how to write simple code. Unfortunately, in most enterprise environments, a significant proportion of the IT workforce seem to enjoy square wheels.

How many times have you seen this?

Some common excuses include it’s too hard, I’m too busy, or I’m not a developer! Often with context around how busy they are working on manual tasks that could be automated, or simplified with a small script.

I won’t tell you that going from no experience to a decent scripter is something you learn overnight, but you need to start somewhere.

PowerShell is a tool that most of us can handle learning, and would benefit from greatly, even right out of the gate. Lower level languages are great for developers, but they might require a more significant investment in time, without as much value in return (for an IT professional).

Let’s illustrate this difference and highlight how helpful PowerShell is, by looking at a simple task across four popular languages.

The Task

We’re going to learn how to read a file. Exciting, right? We’ll do this using the following languages:

PowerShell, a task-oriented scripting and shell language. Work in a Microsoft ecosystem? PowerShell is probably the best way to get things done
Python, a high-level general purpose programming language
C#, a feature-full general purpose programming language
C, a low-level language that terrifies me

This is roughly in order of how much hand holding a language will do for you, which many IT professionals don’t realize. Let’s start from the ground up. Keep in mind there are many ways to skin a cat in each of these languages.

Read a file in C

So! First, we need to get a compiler (Visual Studio is your best bet). Then, we search around for a simple solution. Protip: don’t write in C if you can help it. You need to do everything on your own, which invariably means whatever you write will be insecure.

Here’s a “simple” program to read C:\file.txt:

And voila, it worked!

Good luck querying LDAP, pulling info from VMware vCenter, or hitting a web API using C. Also, get ready for fun if you’re not very familiar with C and need to read or debug someone else’s code.

Let’s move up to a higher level language, C#.

Read a file in C#

Next up, C#. You might have Visual Studio installed already; if not you can compile a simple C# file with csc.exe: e.g. C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe YourFile.cs

Okay! Now we’re getting somewhere. Even if I couldn’t write that on my own, I can get a feel for what’s going on by reading it. But, it’s still a bit heavy on the developer side:

You need to compile
You can’t forget a semi-colon
You’re not just saying “read-a-file”, you have to use .NET classes to tell C# how to read a file. Now do this for every task in your solution. This is why I find comments like just use C# instead of PowerShell quite absurd

If you’re not a developer, it’s important to note how helpful that last bullet is. While C# isn’t a task-oriented language, it uses the .NET Framework, which provides a vast and incredibly valuable library of code.

If you’re an IT professional trying to automate a technology where the vendor got lazy, there’s a good chance you can write a PowerShell function or module that wraps their .NET library (the caveats here still apply: vendors should really provide a PowerShell module for their customers).

Enough rambling, let’s move into the world of scripting, and look at reading a file in Python.

Read a file in Python

So! Being in the Microsoft ecosystem, PowerShell is often the top choice. If you plan to step outside that ecosystem, or just want the experience (protip: this is very valuable), Python is a solid, approachable choice.

We’re going to cut our line count quite a bit:

Nice! So that was much shorter, and the result is even easier to read than C#. Python is a high level language often used for scripting, but it’s not as abstracted out and task-based as PowerShell.

Read a file in PowerShell

We wrap up with the simplest solution:

Get-Content C:\file.txt

It all boils down to a single, easy to read command. If your co-workers complain that they aren’t developers, ask them if they find commands like the one above too difficult. Hopefully they are open to learning, and you can help get them started. Otherwise, what are they doing in IT?

A developer could hop between any of the languages above and more, for a variety of tasks, often with significant expertise in one or two of their favorites. Learning the basics of a task-based language is not too hard, and now that nearly everything has an API, warnings like this will be more than just wishful thinking.

So! What’s the moral of this story anyways?

You Don’t Need to be a Developer to use PowerShell

Anyone in a Microsoft ecosystem should pick up the basics (or beyond) of PowerShell. If you can give another person instructions - maybe read a recipe, or describe how to troubleshoot an iPhone - you already have the basic ingredients of coding. The rest is just syntax, which is a quick Google search away, or even provided to you with snippets.

In fact, PowerShell is a prime candidate for scripting and automation outside of the world of IT. How would we get there?

An open source and cross-platform implementation of the .NET Framework (check) and PowerShell (still waiting)
Microsoft pushing their own product groups to provide an interface through PowerShell, for both administration and working with products (SQL is a prime example), and beyond their Server and Tools organization
Microsoft pushing and assisting third parties in developing PowerShell modules to cover a wider variety of technologies
Microsoft and the community adopting and pushing the PowerShell Gallery - ~560 modules compared to CPAN, PyPI, or RubyGems is sad, regardless of the head start these have
Luck : )

How do I get started?

So! Until then… Are you focused on desktops / clients, printers, identity and access management, databases, virtualization, servers, security, communications, support, networking, or any other pursuit in a Microsoft ecosystem? You should probably learn the basics of PowerShell.

References abound, I always recommend a three pronged approach:

Pick up a formal resource. Maybe a book you read over a month of lunches.
Join the community. Learn from existing tools and scripts. Ask questions and listen in on Twitter, Slack, and PowerShell.org. Stop by a local PowerShell user group. Contribute to all of these if you can!
Use PowerShell every day. Even if only for a few minutes. You can’t learn this in one shot. You won’t learn this if you don’t use it regularly.

Hope to see you around the community!

PowerShell Is Too Hard was originally published by Warren F at Rambling Cookie Monster on February 18, 2016.

Cross Platform PowerShell Remoting

2016-02-06T22:00:00+00:00

Overview

Rambling

Rambling

Wow! Four months since my last post. New city, new home, new job, new tiny human in less than a month, and thankfully, same Wegmans.

I lucked out on the job front. Found a fun team full of talented folks using modern tools, with a role focusing on Microsoft and related tech, in a predominantly *nix / OSS oriented environment. Plenty of room to learn and explore!

Of course, with everyone being well-versed in languages like bash and Python, and with most infrastructure residing on *nix systems using tools like Puppet for configuration, learning PowerShell hasn’t made it to their to-do list. Yet. Hopefully that changes - from my perspective, nothing comes close to PowerShell as an approachable, task-oriented glue language - imagine if it were open sourced and ported to other systems?

Until then, rather than miss out on the goodness PowerShell enables, we need to figure out an approach to invoke PowerShell from cross platform systems!

The Options

There are a number of approaches, each with benefits and drawbacks. At a high level, we could use…

Remoting solutions like SSH or WinRM
API oriented solutions like flancy, or something heavier like ASP.NET Web API
Web based solutions, from tools like rundeck, to shoe-horning Jenkins, to WebCommander, to building your own ASP.NET / C# page

The other options are absolutely worth exploring, but a remoting solution would be very flexible with minimal overhead. So, how can we use WinRM or SSH to call PowerShell from non-Windows systems?

SSH

Wait, an SSH server on Windows? Do you really want to rely on some third party or outdated open source option?

Turns out Microsoft is planning to deliver an SSH client and server for Windows systems, and that there is already a usable (ish) early preview.

Given that it’s so early, you should definitely consider testing this out and contributing feedback. You might help shape design decisions and improve the solution for you, and everyone else using it.

Getting Started with SSH on Windows

This is the easy part! Just follow the instructions here. At some point, this will hopefully receive CI/CD treatment to deliver stable and dev releases, perhaps to a package manager we can hit from OneGet.

If you’re reading this in early 2016, be sure to grab the right architecture; the link in the instructions will point you to the 32-bit release.

Running PowerShell from SSH

I ran into no issues getting up and running with non-interactive commands. Let’s look at a few examples:

# Run ipconfig
ssh -l wframe@my.domain ts1 ipconfig

# PowerShell isn't the default shell. Let's try it out.
ssh -l wframe@my.domain ts1 powershell.exe -command "get-service sshd"

# Double hop issue? Nope!
ssh -l wframe@my.domain ts1 powershell.exe -command "get-aduser wframe"

Nice! So, SSHD is working, we can run PowerShell.exe, and we don’t have to worry about the double-hop issue that you might be familiar with on the PowerShell remoting / Kerberos side!

How about something a little longer, with variables?

ssh -l wframe@rc.domain ts1 powershell.exe -noprofile -command "

    $Service = 'SSHD'
    Get-Service $Service
"

Here’s where we run into our first issue. When calling PowerShell from bash or another interpreter, you need to be mindful of all the interpreters. In our case, bash helpfully replaced $service for us. After bash, presumably cmd.exe needs to be accounted for (i.e. we’re not dropped right into PowerShell), and then PowerShell itself. Maybe you have a snippet that needs double quotes: have fun!

Ideally, you manage Windows systems from Windows systems that are already running PowerShell, and avoid the mess, but you can’t always make that call.

In this example, we can simply escape the $:

ssh -l wframe@rc.domain ts1 powershell.exe -noprofile -command "

    \$Service = 'SSHD'
    Get-Service \$Service
"

Now, you could always abstract this into a shell script, perhaps using expect to handle the password, but ideally we could wrap this in Python or Ruby. I had no luck getting this to work in Paramiko or it’s abstraction, spur, but feel free to give it a try, and be sure to let others know if you have any luck!

That’s about it - let’s move on and look at using WinRM.

WinRb / WinRM

There are a number of ways to invoke PowerShell over WinRM from other languages. For many, the Python and Ruby modules will likely be your first stop.

We’re going to skip the Python pywinrm module, given that it requires either plaintext auth, Kerberos, or fun with SSL. If you already use Kerberos or have a PKI infrastructure in place, you could safely use this. Otherwise, you’ll find a number of guides for pywinrm, WinRb, and more, instructing you to configure WinRM to enable basic auth and AllowUnencrypted. Don’t.

We’ll use the Ruby module for WinRM. Install Ruby on your system, gem install -r winrm, and away we go!

Using WinRM From Ruby

We’ll assume you already have have PowerShell remoting enabled and accessible from another Windows system. Thanks to Dan Wanek and Matt Wrock, we can now use NTLM authentication, which doesn’t come with the dependencies or complexities of Kerberos or a PKI.

Security note: Do consider using SSL and/or a more modern authentication mechanism like Kerberos, but for quick and dirty testing, NTLM is a far safer bet than basic auth with AllowUnencrypted. Would love to hear some real security folks weigh in on the current method…

Let’s get started!

Here’s the gist of what I might run. More examples abound, just keep in mind we’re using create_executor’s run_powershell_script method, not the deprecated alternatives.

#!/usr/local/bin/ruby
require 'winrm'
winrm = WinRM::WinRMWebService.new('http://ts1:5985/wsman',
                                   :negotiate,
                                   :user => 'wframe',
                                   :pass => 'my password')

winrm.create_executor do |executor|
  executor.run_powershell_script("get-service 'sshd'") do |stdout, stderr|
    STDOUT.print stdout
    STDERR.print stderr
  end
end

This is my first time touching Ruby outside of some sparing use with the GitHub pages setup behind this blog, but I figured it would be worth trying to abstract this into a simple, re-usable function. Cobbled this together from various stackoverflow queries; feedback or tips welcome.

Let’s kick the tires a bit:

# Double hop issue? Yep!
ps-winrm.rb -u wframe -c "get-aduser wframe"

The error message is a bit garbled, but you can get the gist of it. While the underlying cause differs, we run into the double hop issue PowerShell remoting typically gives us.

Given that NTLM support was just implemented, and that a cursory search doesn’t turn up anything on CredSSP, it looks like the usual CredSSP workaround won’t help.

Security note: CredSSP will make a red team happy. Do consider the security implications, as well as the fact that if you’re using RDP, you’re already using CredSSP.

Another workaround that works here would be to live dangerously and hit a domain controller directly:

That’s about it! I have seen some odd issues with commands that use runspaces, but most every-day use cases seem to work.

Which Should I Use?

Both options have their strengths and weaknesses, this is really up to you. I personally prefer to use languages like Ruby or Python over bash, and SSHD is quite new, so I’ll likely lean towards WinRM in the interim. Once I can get Paramiko or spur working against Microsoft’s SSHD, I’ll likely switch over to that.

Looking ahead, SSH on Windows seems very promising, and should enable simpler integration with cross platform solutions.

The WinRM implementations are worth exploring and will continue improving, but will come with some limitations, and don’t seem to be receiving contributions from Microsoft (yet).

What’s on the Horizon?

A nice side-effect of Microsoft using GitHub is that we get a glimpse of what’s coming down the road.

Check out the Win32-OpenSSH issues - you’ll see some notes on current limitations and bugs, along with some interesting ideas and plans. Curious to see how they’ll implement key based authentication for domain accounts (issue 39).

Be sure to kick the wheels of Microsoft’s SSHD implementation. You can actually contribute, whether you submit a pull request, a thoughtful bug report, or an idea for a feature. It’s still quite early, so if you get in your ideas now, who knows, you might help improve this for everyone!

Cheers!

Cross Platform PowerShell Remoting was originally published by Warren F at Rambling Cookie Monster on February 06, 2016.

Finding Evil LDAP Queries

2015-10-02T22:00:00+00:00

Overview

Rambling
The Fun Begins
Finding Expensive, Inefficient, or Long Running LDAP Queries
PowerShell Modules: Write Them
Get On With It!

Rambling

A while back, I noticed an intriguing tweet from Mark Morowczynski at Microsoft:

Long story short, Mark explains how to search for and analyze expensive, inefficient, and long running LDAP queries in Active Directory. Bookmarked.

This week, Mark’s article came in quite handy.

The Fun Begins

Wednesday morning, someone asked about some SCOM alerts for AD. We see these every so often, and I started explaining that sites with one or two domain controllers lead to somewhat common and often innocuous AD Site Performance Health Degraded alerts.

I don’t like assumptions though. Interesting. All the domain controllers in one site are at 100% CPU, with crazy CPU queues. This wasn’t a site getting disconnected, or a one-off evil LDAP query. Looks like we get to dive into Mark’s post!

Finding Expensive, Inefficient, or Long Running LDAP Queries

Prerequisites

Okay! I know I saw ‘PowerShell’ when skimming the article, this should be quick and easy, or so I thought.

The final stages of our 2012 R2 roll out are scheduled for next week, so we need to install KB2800945 on any remaining 2008 R2 domain controllers, and reboot.

AD is somewhat important, and a juicy target - if you have a tight ship and this is a rare scenario, consider opening a ticket with Microsoft and triggering your (security) incident response process here.

The Registry

Once our prerequisites are in place, we need to flip a few values in the registry.

Oh. There’s a screen shot of the registry in the blog post. And a table with registry keys and values. Guess we’ll write a PowerShell function!

I’m all for quickly fighting fires with a GUI, but I realize I don’t want to deal with enabling and disabling this manually on several machines. Writing a quick POC function will be far faster. Particularly given that this is a simple registry change, and that Shay Levy has written the fantastic PSRemoteRegistry, which I no longer consider a dependency. Just say no to providers that don’t let you hit remote systems.

All we do here is enable logging, and set a few parameters that Mark suggested.

Set-RegDWord -ComputerName $Computer -Hive LocalMachine -Key 'System\CurrentControlSet\Services\NTDS\Diagnostics' -Value '15 Field Engineering' -data 5
Set-RegDWord -ComputerName $Computer -Hive LocalMachine -Key 'SYSTEM\CurrentControlSet\Services\NTDS\Parameters' -Value 'Expensive Search Results Threshold' -data 0
Set-RegDWord -ComputerName $Computer -Hive LocalMachine -Key 'SYSTEM\CurrentControlSet\Services\NTDS\Parameters' -Value 'Inefficient Search Results Threshold' -data 0
Set-RegDWord -ComputerName $Computer -Hive LocalMachine -Key 'SYSTEM\CurrentControlSet\Services\NTDS\Parameters' -Value 'Search Time Threshold (msecs)' -data 100

That’s it! My domain controllers are now logging event 1644, with details on each LDAP query that meets the thresholds I just set; in this case, anything taking over 100ms. Time for coffee.

Collecting the Results

This could be outdated or flat out ignorant knowledge, but I recall wevtutil epl being incredibly fast, as compared to reading events using Get-WinEvent (or Get-EventLog, eek!). I’m a fan of using tools that meet your needs. In this case, quickly exporting an evtx is a simple one liner.

Invoke-Command -ComputerName $Computer {wevtutil epl 'Directory Service' "\\$Using:Computer\c$\$ENV:ComputerName-Evil.evtx"}

Alrighty, the events are there. Oh. Be sure your Directory Services event log is large enough to catch enough data. Maybe use DSC.

At this point, I reverse the registry changes I made, and get ready for some fun with Excel!

Parsing and Interpreting the Results

Mark links to a script that helps read Event 1644 data. This is quite handy, and it’s awesome that folks are trying to use PowerShell, but the methodology leaves a bit to be desired.

Install Excel, so that you can access the COM interface. Sigh.
Read the script and Get-Help. No parameters. Sigh.
Run the script, follow the Read-Host guide. Sigh.
If you’re like many folks out there, you might get a few errors. My output was still usable.

At this point, you can browse around the results in Excel, as Mark explains in his post.

In our case, the TopTime-IP section came in handy. A single system is making a huge portion of these queries, including a number of 8+ second queries using UID, which isn’t indexed. I gleefully brandish my pitchfork and run a reverse lookup.

Hopefully this pointed you in the right direction or right at the culprit. Depending on what you do or don’t find, there are a variety of helpful resources:

Indexing. Active Directory sits on a database. Like most database engines, your domain controllers will be much, much happier if queries use an indexed attribute.
Developers, vendors, and sysadmins might not be familiar with how evil the queries they write can be. Long story short? Be as specific as possible. Start as deep as you can, set your search scope as tight as possible, and follow efficient query practices like using indexed attributes, returning only the attributes you need, and keeping your wild cards at the end of a string, not the start.
Other tools, like the AD data collector sets, SPA, WireShark, or Microsoft Message Analyzer might come in handy.
We’re assuming you ruled out other common causes, like non-LSASS offenders, mis-sized domain controllers, and other scenarios. Google and MSFT resources are your friend : )

Now, what about next time?

PowerShell Modules: Write Them

Abstraction is important. Don’t think in terms of “writing scripts”, try to think in terms of writing re-usable tools: functions. Once you’re comfy writing these tools, you need a toolbox to put them in: modules.

Want to forget about tools and toolboxes to an extent? Desired State Configuration and configuration management solutions that layer on top might be up your alley. But I digress, and it’s important to note that configuration management isn’t a panacea; Do you plan to change your configurations for a class of system just to enable diagnostics on a handful of instances? Nope.

I used a quick recipe for building a PowerShell module, published the obnoxiously-named PSLDAPQueryLogging module to GitHub and the PowerShell Gallery, and I’m ready for next time!

Here’s a quick snippet where I enable logging on all domain controllers, pull back the logs, and disable logging:

Get On With It!

Be sure to share your modules on GitHub and the PowerShell Gallery. Languages like Perl, Python, and Ruby all have repositories like this, each with thousands and thousands of modules. We have 321. Let’s get to work!

Barring the occasional I have to share it! discoveries, this will likely be my last post for some time; life is about to get crazy at home. In a good way. Cheers!

Finding Evil LDAP Queries was originally published by Warren F at Rambling Cookie Monster on October 02, 2015.

Building a PowerShell Module

2016-07-24T05:30:00+00:00

Overview

Rambling
Why Modules?
This Seems Complicated!
The Ingredients
The Recipe
Following the Recipe
How I Write Modules, Summarized
PSStackExchange
Wrapping Up
- Side Note for Vendors

Rambling

A while back, someone mentioned it might be fun to document the PowerShell-module-writing-process. This has been done before, but I figure it would be fun to post my own process, from initial idea through publication on the official PowerShellGallery.com site.

I recently discussed the August Scripting Games puzzle on PowerShell.org, which involved querying a web API. It turns out this is a very common need, and many of the modules we write abstract out these APIs into handy PowerShell functions and modules.

We’re going to make the assumption you know what a module is, and that you have some experience writing PowerShell functions. If not, be sure to spend some time learning PowerShell before continuing here!

This post will cover my typical formula for writing a module, using the Stack Exchange API as an example. Feel free to browse the PSStackExchange code on your own.

Why Modules?

Advanced functions will take you far with PowerShell. If you aren’t writing functions today, be sure to start encapsulating your code in these re-usable tools. But… they have their limits. Here are a few reasons you might bundle your advanced functions in a module:

Simplify code organization
Group related functions together
Share state between functions, but not with the user
Re-use “helper functions” that you don’t want exposed to the user
Improve discoverability: Find-Module MyModule or Get-Command -Module MyModule
Simplify distribution: Install-Module MyModule

In our example, we will organize a set of Stack Exchange functions into one module.

This Seems Complicated!

Doing this from scratch might take you a little time. Thankfully, once you write a module or two, you can quickly get started by copying it and tweaking a few files. Don’t be scared off by the length of this post; writing your own modules is well worth spending a few minutes to pick up the basics!

The Ingredients

There are many ways to create a module, from slapping a .psm1 extension onto a file, to compiling a fully fledged binary module from C#. We’ll take a common middle ground here, and use the following ingredients:

A Module Manifest. This is a .psd1 file that describes your module. PSStackExchange.psd1
A Root Module. In our case, a script module .psm1 file. This is just PowerShell code to run when importing the module. PSStackExchange.psm1
Exported (Public) Functions. These are the advanced functions an end user can run from our module. For example, Get-SEQuestion.ps1 or Get-SEObject.ps1
Private Functions. These are optional “helper functions” that we want to use in our exported functions, that the end user shouldn’t see. For example, Add-ObjectDetail.ps1 or Join-Parts.ps1
Formats. These are optional format.ps1xml formats to help decorate your output, often specified in the module manifest ‘FormatsToProcess’. PSStackExchange.Format.ps1xml
Readme. If you’re using GitHub or another common code repository, the Readme.md is a handy front page for your project, written using simple Markdown rather than HTML
AppVeyor config. If you’re using a supported version control solution, AppVeyor enables simple and free continuous integration and delivery for open source projects. AppVeyor.yml

We have our ingredients, let’s look at a recipe for a module!

The Recipe

We’re going to do this in a few quick steps:

Create a GitHub repository
Create the module and scaffolding around it
Hook up AppVeyor and publish the module

This might take a few minutes the first time you run through it, but you can borrow and tweak this same scaffolding for each module you write. In fact, you might find or write helper PowerShell modules and tools that simplify this process.

Let’s get to work!

Following the Recipe

There’s no real order to this; depending on what you do or don’t incorporate, don’t feel like you need to follow this to the letter.

Create a GitHub Repository.

This should be pretty straightforward. If you haven’t used GitHub before, the following might help:

All we need to do is:

Create an account on GitHub, download GitHub for Windows
Create a new repository (We’ll call it PSStackExchange, and pick the MIT license)
Clone PSStackExchange using GitHub for Windows

Let’s move on to the most important bit, the module itself.

Create the Module and Scaffolding Around It

Here’s how I typically organize my modules. We’ll use PSStackExchange as an example, substitute this out for your own module!

PSStackExchange\
- en-US\ (or locales of choice)
  - about_PSStackExchange.help.txt
- Private
  - Join-Parts.ps1
  - Get-SEData.ps1
  - …
- Public
  - Get-SEObject.ps1
  - Search-SEQuestion.ps1
  - …
- lib\ (Not used in this module)
  - Some.Library.dll
- bin\ (Not used in this module)
  - SomeDependency.exe
- PSStackExchange.Format.ps1xml
- PSStackExchange.psd1
- PSStackExchange.psm1

If we’re going to be adding our project to GitHub or a similar code repository, we add a little more scaffolding:

Repository Root
- PSStackExchange\ (Module folder described above)
- Tests
  - PSStackExchange.Tests.ps1
  - Appveyor.Pester.ps1
- README.md
- AppVeyor.yml

I ran through the following code to get started. Typically I’ll just copy the scaffolding from another module, create a new GUID in the psd1, and tweak other module specific references.

In our case, we have a few Stack Exchange advanced functions that hopefully follow a few best practices, some private helper functions that we don’t want the user to see, and a few other files to cover testing and usability.

In PSStackExchange.psm1 we load our public and private functions. If a module is a work-in-progress, I’ll usually export $Public.Basename to avoid hard coding functions to export in the psd1. Once a module is released, I try to add the public functions to the psd1.

If you’re writing a module, you should consider writing Pester tests for it. It’s quite comforting to have a suite of tests that run automatically after each change you push, rather than assuming the code you write was correct, or attempting to manually test your code after each change. Give it a shot! We include a few superficial tests in PSStackExchange.Tests.ps1.

Lastly, we include some usability features. We add an about_PSStackExchange help topic, we decorate our output with the PSStackExchange.Format.ps1xml file, and we add some notes on how to install and use the module in the README.md.

We’re good to go! Let’s look at how we can publish this module for others to use and improve.

Hook up AppVeyor and Publish the Module

The content of our module is ready to publish. Before we publish this, we’ll enable continuous integration with some handy automated testing through AppVeyor.

First, we set up our project in AppVeyor by adding appveyor.yml to the repository, and adding the GitHub project to our AppVeyor account. We abstract out the calls to Pester in AppVeyor.Pester.ps1, using some ideas from here.

Next, we push the changes we’ve made on our computer up to GitHub. Our code is now published, and AppVeyor will start running a build.

Lastly, we want to publish our module in the PowerShell Gallery, giving end users with PowerShell 5 a simple way to find and install your module. We could hook this up to automatically run in AppVeyor, but that’s a topic for later.

Sign on to PowerShellGallery.com with your Microsoft account
Get your API key (find it here)
Publish your module!

Our module is now live on PowerShell Gallery!

How I Write Modules, Summarized

Whew! That was a long post. Thankfully, most of this stuff can be re-used in each module you write. Let’s review the steps:

Create a GitHub repository
Create the module and scaffolding around it
Hook up AppVeyor and publish the module

The first and last step take a minute or two each. The module and scaffolding around it can be copied and tweaked, which should only take a few minutes. Most of your time will be spent writing the advanced functions for the module.

PSStackExchange

The module is published and ready to use! I’m on another computer with PowerShell 5, I can get up and running with a few lines of code:

Here’s some output from the examples:

Wrapping Up

That’s about it! If you aren’t writing modules already, you should definitely consider it. Looking for further reading? Here are a few references that might come in handy:

Module Design Rules - This is from the 2014 PowerShell Summit, thanks to Kirk Freiheit
Further Down the Rabbit Hole: PowerShell Modules and Encapsulation
How to Write a Module Manifest
Windows PowerShell Modules
about_Modules
Building PowerShell Functions - Best Practices - Shameless plug. Includes a number of references.
Learn PowerShell Toolmaking in a Month of Lunches
Free eBook on PowerShell Advanced Functions
The PowerShell Best Practices and Style Guide
Monad Manifesto - This gives a nice overview of the vision and goals set out for PowerShell. If you’re writing modules for public consumption, consider reading this, to avoid publishing something as awful as Citrix’ PVS “PowerShell” snapin.

Side Note for Vendors

Writing PSStackExchange reminded me how important it is for vendors of enterprise products to provide PowerShell modules that wrap their product’s API. Despite a nice API and decent documentation, writing a feature-poor PowerShell module for this was just as painful as wrapping the Infoblox API.

Vendors: If your competition provides a PowerShell module and you do not, there’s a good chance I’ll push for your competitor’s product. This is a major value-add if you do it right and follow PowerShell conventions.

Cheers!

EDIT July 2016: Updated links to PSStackExchange to link to a specific point in time. The current version of this project may see updates to illustrate things like PSDeploy.

Building a PowerShell Module was originally published by Warren F at Rambling Cookie Monster on September 06, 2015.

PowerShell Configuration Data

2015-08-16T22:00:00+00:00

Overview

Rambling

I recently wrote PSDeploy, a quick-and-dirty module to abstract out PowerShell based deployments. You define what you want deployed in a configuration file, it does the rest.

The most common follow-up questions seemed to be why yaml? or why not <data format of preference>? This is a quick hit on the many data formats you can use from PowerShell.

Disclaimer: I know very little about data formats, their intended uses, or their benefits or caveats. This is from a layman’s perspective.

The Choices

Let’s list off a few common data formats we could use.

XML
JSON
YAML
PowerShell Data File (PSD1)
INI
Registry
CSV
Text
Database

Let’s take a peak at these from a high level. Keep in mind these can vary wildly even within a single format. For example, you might have an ad hoc JSON file, or a formal schema describing your JSON.

For each solution you design, consider your needs, priorities, and constraints, and pair them up with one of these formats. Bear in mind the the lessons of the configuration complexity clock.

XML

XML is a tried and true format, but is a bit inefficient and ugly.

Common tools

Built in Cmdlets: Import-CliXml, Export-CliXml, ConvertTo-Xml
Built in .NET framework support

The Good

Avoid external dependencies
Flexible
Wide cross-platform support and tooling
CliXml functions provide simple serialization and deserialization
Stores data with more than one layer of depth

The Bad

Not human readable
Syntax is inefficient and verbose
It’s 2015

Example

# Serialize some data to disk
Get-Item C:\Windows\explorer.exe |
    Export-Clixml -Depth 5 -Path C:\XML.xml

# Deserialize the data
$File = Import-Clixml -Path C:\XML.xml

# Drill down.
$File.VersionInfo.ProductVersion

Should I use XML?

There are two scenarios where I use XML. In all other cases I pick an alternative.

Quick and dirty serialization. Import and Export CliXml are simple to use
Technology lock-in. If it only supports XML, you don’t have a choice

JSON

JSON is a lightweight data format common in modern web APIs.

Common tools

Built in Cmdlets: ConvertFrom-Json, ConvertTo-Json (PowerShell 3 or later)

The Good

Avoid external dependencies (in PowerShell 3 or later)
Stores data with more than one layer of depth
Semi human readable
Syntax is more efficient and less verbose than XML
Implemented in libraries across several languages

The Bad

Want to store MSFT paths? Have fun: { "Path": "C:\\W\\T\\F" }
Not as widespread cross-platform support or tooling as XML

Example

# Convert some data to Json
$JSON = Get-Item C:\Windows\explorer.exe |
    ConvertTo-Json -Depth 2

# Read the Json back into an object
$File = $JSON | ConvertFrom-Json

# Drill down.
$File.VersionInfo.ProductVersion

Should I use JSON?

JSON is a popular and safe choice nowadays. Depending on your needs, this is often a good fit.

YAML

YAML is a human friendly data format.

Common tools

PowerYaml and various forks
Yaml.Net

The Good

Very human readable
Stores data with more than one layer of depth
Syntax is more efficient and less verbose than XML
Implemented in libraries across several languages

The Bad

Dependencies on libraries like Yaml.Net
Haven’t seen a reliable serialize and deserialize module for YAML yet
Whitespace is part of the syntax

Example

# Define some Yaml
$Yaml = @"
Name: explorer.exe
Length: 4532304
DirectoryName: C:\Windows
VersionInfo:
  ProductVersion: 10.0.10240.16384
"@

#Read the Yaml, using PowerYaml
$File = Get-Yaml -FromString $Yaml

# Drill down.
$File.VersionInfo.ProductVersion

Should I use YAML?

I would only recommend YAML where human readability is paramount, and your other needs and constraints don’t rule it out.

Yaml a great option when you will simply be reading in a config file, and the end users will be manually manipulating this file.

PowerShell Data File (PSD1)

PowerShell data files are used for PowerShell module manifests, but can be used to store arbitrary data.

Common tools

Built in Cmdlet: Import-LocalizedData

The Good

Familiar to PowerShell authors
Avoid external dependencies
Stores data with more than one layer of depth
Semi human readable

The Bad

Single platform
Haven’t seen a reliable serialize and deserialize module for PSD1 files yet

Example

# Create a PSD1 file
@"
@{
    Name = 'explorer.exe'
    Length = 4532304
    DirectoryName = 'C:\Windows'
    VersionInfo = @{
      ProductVersion = '10.0.10240.16384'
    }
}
"@ | Out-File -FilePath C:\PSD1.psd1

# Read the file
$File = Import-LocalizedData -BaseDirectory C:\ -FileName PSD1.psd1

# Drill down.
$File.VersionInfo.ProductVersion

Should I use PSD1?

This seems like a reasonable choice for PowerShell configuration files that will be edited by hand, by folks familiar with PowerShell.

Other Data Formats

There are a variety of other choices. Here are a few others you might consider:

CSV has plenty of built in Cmdlets, but is quite limited and might produce unexpected results.
Text is an option, with a variety of tools including the new ConvertFrom-String. Not sure why you would chose this over an existing data format.
The registry. I wouldn’t pick it, but it’s used by many applications and is familiar to most Windows administrators. Tools like PSRemoteRegistry make this simple to work with remotely, unlike the registry PSProvider.
Ini files are a bit dated, but are simple to read and use. Given their limitations and the wealth of other options, you should probably leave these in the attic.
Databases are a great option for larger solutions, or where your data model requires a bit more sophistication. SQLite is a handy, cross-platform solution, if SQL Server or other database engines are too heavy-weight.

What Should I Use?

Like most answers in the world of computing: It depends. Chances are, even with your particular needs and constraints, there isn’t one correct choice. Consider your options, and pick a data format that makes sense to you.

Do you enjoy trolling? Consider JSONx:

Cheers!

PowerShell Configuration Data was originally published by Warren F at Rambling Cookie Monster on August 16, 2015.

PSDeploy: Simplified PowerShell Based Deployments

2015-08-15T22:00:00+00:00

Overview

Rambling
Continuous Deployment
PSDeploy
Example Deployment Scenarios
- Basic Filesystem Deployment
- FilesystemRemote Deployment
Extending PSDeploy
Next Steps

Rambling

A short while back I gave my first live webinar, on getting started with Git and GitHub. It was both exciting and a bit terrifying. I’m gearing up for similar sessions at work to get folks going with our version control solution, Stash.

When I look at version control as an IT professional, one pain point that we could help with stands out. In addition to changing their workflow and learning version control, folks are now working with files and folders in source control. Not where they live. Wouldn’t it be handy if we could use version control, and not worry about remembering to be sure to move the files to their appropriate homes?

This post will discuss a simple PowerShell based deployment solution: PSDeploy.

Continuous Deployment

Developers realized this was important long ago. There are some specialized deployment tools like Octopus Deploy, continuous integration and deployment tools like AppVeyor or TeamCity, and continuous integration tools like Jenkins that can be shoehorned into providing deployments.

So, with all these solutions, why duct-tape something together?

I like fun side projects
A simplified deployment system might help with buy in for our version control solution
We use Jenkins. Their build definition process leaves a bit to be desired. This allows me to use the same abstracted build script for every project, as long as I have a deployments.yml in the repo root
This is portable, and might help folks who use more than one tool chain. I could use PSDeploy with Jenkins, TeamCity, AppVeyor, Bamboo, etc.

Let’s dive in.

PSDeploy

This is a quick and dirty module that simplifies deployments. You create a deployment config file, PSDeploy reads it and runs your deployment(s).

Ingredients

YAML: Data format for deployment config files. Even easier to read than JSON. Familiar to folks using tools like AppVeyor.
Deployment config: YAML files defining what is being deployed. They should have a source, a destination, a deployment type, and might contain freeform deployment options.
Deployment type: These define how to actually deploy something. Each type is associated with a script. We’re including FileSystem and FileSystemRemote to start, but this is extensible.
Deployment script: These are scripts associated with a particular deployment type. All should accept a ‘Deployment’ parameter. For example, the FileSystem script uses robocopy and copy-item to deploy folders and files, respectively.

A Simple Illustration

Let’s looks at an example deployments.yml and the outcome of invoking it.

Here’s a project I’m working on. I want to deploy MyModule to a few paths every time I push a commit. I should probably gate this with Pester tests as well : )

Here are the content of the deployments.yml:

MyModuleDeployment:
  Author: 'wframe'
  Source:
    - 'MyModule'
  Destination:
    - 'C:\Temp\MyModule'
    - '\\C-IS-TS-91\C$\Users\wframem\documents\WindowsPowerShell\Modules\MyModule'
  DeploymentType: Filesystem
  Options:
    Mirror: True

We can verify how PSDeploy will parse this:

Let’s invoke the deployment:

If we check the destination paths, we find MyModule has been deployed!

What happened under the hood?

We read the deployments.yml
We see that the source is a folder
We see that the deployment type is Filesystem
We look up the script for filesystem deployments
We execute the filesystem script for this deployment
The script reads the deployment and processes it:
- It’s a folder, so we use robocopy /E /XO
- The ‘mirror’ deployment option is set, so we add on /PURGE

If you prefer illustrations, here’s quick diagram of a similar deployment:

That’s pretty much it! Instead of defining all this code in my continuous deployment system, I just run Invoke-PSDeployment.

Getting Started

First off, download and explore the module:

Running through the code above, we see a few commands you can work with:

Get-PSDeployment: Read a deployment config file
Get-PSDeploymentType: Get details on a deployment type
Get-PSDeploymentScript: Show available deployment types and associated scripts
Invoke-PSDeployment: Run a deployment

Let’s put this to work in a few real world scenarios.

Example Deployment Scenarios

Basic Filesystem Deployment

I want to deploy C:\Git\MyModuleRepo\MyModule to \\Server\Modules. How can I use PSDeploy to do this?

First, we add a config file, C:\Git\MyModuleRepo\deployments.yml. Here’s the content:

MyModuleDeployment:
  Author: 'wframe'
  Source:
    - 'MyModule'
  Destination:
    - '\\Server\Modules\MyModule'
  DeploymentType: Filesystem
  Options:
    Mirror: True

We verify that the deployment will do what we want:

Looks good to me! In our continuous deployment solution, we simply run this:

    Invoke-PSDeployment -Path 'C:\Git\MyModuleRepo\deployments.yml'

This gist shows how you can test out PSDeploy without setting up any fancy tools. All you do is create a dummy folder to deploy, targets to deploy it to, and invoke the deployment.

Let’s look at a more involved deployment.

FilesystemRemote Deployment

So! I’m using Jenkins for continuous integration and trying to shoehorn in a deployment. I’m running into trouble: Jenkins is running with an account that doesn’t have the right privileges to deploy to the targets.

We can use the FilesystemRemote deployment type, which will run the robocopy and copy-item commands in a PowerShell remoting session with appropriate credentials. I know, it’s silly, pull requests for a more appropriate solution would be welcome : )

Ahead of time, I locked down the Jenkins server and set up EnvInject, with a little help from Matt Hodgkins’ post.

Next, I set up a project with the following PowerShell build command. I’m omitting the testing steps for simplicity. You should strongly consider gating your deployments using Pester tests.

This looks a bit complicated, but all we’re doing is loading up PSDeploy, grabbing credentials for our deployment, and kicking off the deployment with some parameters. You can run Get-Help Invoke-PSDeployment -Parameter DeploymentParameters for more information on passing parameters to a deployment script.

That’s about it! Maybe SomeSessionConfig is a delegated endpoint, which works around the double hop issue. I could also configure a CredSSP endpoint, but that might draw the wrath of the security community. Tip: Do consider the risks of CredSSP, but if you still have admins using RDP, you’re already using CredSSP, and you have bigger problems to worry about : )

We just walked through a few example deployment types. What if you want more deployment options?

Extending PSDeploy

I’m probably getting ahead of myself, but I tried to make PSDeploy somewhat extensible. Follow these steps to add a new deployment type:

Update PSDeploy.yml (Associates scripts to deployment types)
Write the deployment script
Consider whether and where to store options in the deployment config

Update PSDeploy.yml

This file is stored in the root of the module, although you can move it to a central spot. It stores the associations of deployment types to deployment scripts.

The deployment type is the root node.
The script node defines what script to run for these deployment types
The description is exposed when running Get-PSDeploymentType

For example, I might add support for SCP deployments:

SCP:
  Script: SCP.ps1
  Description: Deploys artifacts using SCP. Requires Posh-SSH

Create the Deployment Script

These are stored in the PSDeploy folder under PSDeployScripts, although you can point to other paths in PSDeploy.yml

Following the SCP example, our deployment script might be C:\Path\To\PSDeploy\PSDeployScripts\SCP.ps1
In your deployment script (SCP.ps1), include a ‘Deployment’ parameter
- See FilesystemRemote.ps1 for an example
Include comment based help to explain the deployment and any extra parameters you accept. This is a good spot to describe YAML options, if you use them.

Here’s how I implement the deployment parameter in the filesystem deployment scripts:

YAML Options

If you want to include options in a deployment config file, similar to the mirror option we use in Filesystem deployments, just include them in deployments.yml.

Get-PSDeployment converts the deployment config YAML and processes it into a number of ‘deployment’ objects that are passed to your script.
No special steps are needed for the deployment config, just add nodes as needed.
- We extract the YAML Options node with Get-PSDeployment, and process it into a DeploymentOptions property on the Deployment object
- If you need even more flexibility, we store the raw converted YAML in the Raw property returned from Get-PSDeployment

Here’s an example extended deployment config with a hostname and port option:

SomeExtendedDeployment:
  Author: 'wframe'
  Source:
    - 'SomeSourceFolder'
  Destination:
    - '/Some/Target/Path'
  DeploymentType: SCP
  Options:
    Hostname: Server1
    Port: 22

We can explore this with Get-PSDeployment to see what we would work with in out deployment script:

Next Steps

That’s pretty much it! I’m hoping that we get more IT professionals using version control, continuous integration, continuous delivery, and similar solutions. We might see more (and better!) tools to simplify and streamline using these solutions as an IT professional.

Don’t be afraid of tools and ideas traditionally associated with developers or ‘DevOps’. Even if you work in an enterprise that has few if any developers, these tools and the ideas behind them can benefit IT as a whole.

If you haven’t already, you should consider joining the open source community. Suggestions, pull requests, and other contributions to PSDeploy would be more than welcome!

PSDeploy: Simplified PowerShell Based Deployments was originally published by Warren F at Rambling Cookie Monster on August 15, 2015.