This is a short one, and an obvious one, but I feel like I should have absorbed it much earlier in my "I'm not an SRE I just need to get my problem solved" career.

The other day I needed to run some command on a production box. The command does some stuff, then spits out a handful of CSVs including records of what it did. You know, for record keeping.

So I logged onto the box, ran my_command, pulled out the three CSVs by hand with some kubectl cp calls, and called it a day.

A day later somebody comes to me and asks me to run the command on ~100 boxes, and to collect the records.

Running the command itself on a bunch of boxes is easy enough. I can just use our internal management tool, do internal-box-mgmt-tool command -f some-filter my_command, and that'll run the command across the boxes matching some-filter. Including nice concurrency control and log collection, so I get box-1.log, box-2.log on my machine as a result.

But this person really wants all the CSVs. For record keeping. But the management tool really just runs commands and then shuts down the whole endeavor. No collection of created files, no passing go, no collecting of $200.

my_command is a wonderful command. It was reviewed by my peers, properly checked into the repo, and does the work. But it writes to disk. I just needed a bit more, and really didn't want to go through the motions of the review + getting it out into the world.

After asking several people and having them all say "just rewrite the command to write to S3", I sat with my grief, denying I should "just" do the right thing.

Finally I landed on what I had to do:

my_command
echo "YYY 1"
cat widgets.csv
echo "YYY 2"
cat thingies.csv
echo "YYY 3"
cat stuff.csv
echo "YYY 4"

Since the management tool would save my stdout logs, I just threw the file contents into the logs.

(In practice my command became internal-box-mgmt-tool command -f some-filter "my_command; echo 'YYY 1'; cat widgets.csv; echo 'YYY 2'; cat thingies.csv; echo 'YYY 3'; cat stuff.csv; echo 'YYY 4'" -o logs/)

Over in Python-land, to process the logs "back" into the report:

results = Path("results")
for log_file in Path("logs").iterdir():
  # results/box-1, results/box-2, etc
  box_results = Path("results") / log_file.name

  log = log_file.read_text()
  _, report_1, log = log.split("YYY 1", maxsplit=2)
  report_2, log = log.split("YYY 2", maxsplit=1)
  report_3, _ = log.split("YYY 3", maxsplit=1)
  # added bonus of trailer: making sure we've seen
  # all the output
  report_3, _ = report_3.split("YYY 4", maxsplit=1)_
  (box_results / "widgets.csv").write_text(
    report_1
  )
  (box_results / "thingies.csv").write_text(
    report_2
  )
  (box_results / "stuff.csv").write_text(
    report_3
  )

Sometimes the nail meets a hammer. And in the end the "nasty" solution only ends up taking 20 lines.

Yet again I am reminded of the importance of acting like an operator when doing operations.