Running a Cygwin Bash Script in Task Scheduler

 Familiar readers may know the context. I have a textfile full of famous/other quotations, one per line. I deployed a freeware product Qliner Quotes which provided a way of generating HTML/other format signature formats which could be used to select a random quote from my custom or other quote files snd to attach the filled boilerplate to the bottom of my Thunderbird emails. I later basically wrote a bash script which could separately fill the html boilerplate file and also would read my latest quote html file into a browser tab and use it to headline my latest daily miscellany political post. I ran the bash script in Windows' Ubuntu interface, later putting it into Windows Task Scheduler to generate a new quote html file every 30 minutes. No major problems but it helped set my expectations into performance.

I'm not sure exactly what happened but probably recent Windows patching caused a functionality problem. I know there was a recent Linux-related patch, but my issue predated that implementation. I recall 2 or 3 weeks back updating my Ubuntu stack. So my scheduled task was no longer working, and I decided to install or reinstall cygwin. (I may have last reinstalled cygwin on an earlier PC.) There were various practical uses for using cygwin, like dealing with naming conflicts in merging file directories, attaching timestamps to certain files, etc. Yes, there are freeware renaming utilities, etc.; Windows has powershell, etc. But there are things I can do in seconds based on writing Unix scripts for over 20 years. I had to slightly tweak the bash script I had written for Windows linux subsystem but no major effort.

What I didn't expect were issues implementing my cygwin shell script into Task Scheduler. There were minor tweaks (like dealing with embedded spaces in directory paths, the arguments to the bash command to use, i.e., -c -l, specification of the target script reference, etc.) But one of the most irritating things was how long it took for the job to run out of the scheduler, e.g., up to an hour or longer vs. several seconds in a regular bash session. While I could launch a bash session if and when I wanted to rotate quotation html files used in emails and blog posts, I don't recall running into these kinds of delays when I was running a similar script in Task Scheduler with the Windows integration of Ubuntu, and I didn't want to  launch a session just to run a script.

I finally got the performance issue resolved using a tip from the third source below: export the task xml file from Task Scheduler, edit the priority from 7 to 4, drop the old task in Scheduler, and import the edited xml file back into Task Scheduler. Now if and when my bash session launches every half hour, the session completes within several seconds.

Note: I found the following sources useful in troubleshooting my issues: 1, 2, 3.

Robocopy and a Weird Quirk in its Utilization

 As a professional (Oracle) database administrator, I'm obsessed with backups, including for my own personal computing. I have a strategy that includes backups across applications (e.g., a permanent email repository for my email client), multiple PC's/external drives/flash drives and multiple cloud storage vendors. Some might call me a digital packrat of sorts. But it does have its positive sides; I occasionally needed to pull up an old proof of payment, my updated Security+ certificate, a software utility I can no longer find online, etc. I have a collection of pdfs, certain video clips, political cartoons, embedded charts, etc. used in my signature political blog.

So frequently I have a cloud source directory including things like email files/folders, podcasts, etc. And usually I want to synchronize these files. Obviously I want to propagate the latest version of my resume and workfiles, emails, etc. On the other hand, once I've heard one of the podcast clips and delete it (I sometimes save a noteworthy episode, but that's the exception), I no longer need the old backup copy.

There are synchronization software products (I still have one or 2 installed on my workhorse PC), but occasionally I ran into usability issues (like the process getting hosed in the middle of a lengthy synchronization). I don't remember when exactly I learned about the Windows utility robocopy (certainly an improvement over my writing complicated XCOPY commands and the like), but I worked with some Windows administrators at work maybe 4 years back who heavily relied on it, and I've used it since then to some extent.

So suppose my C drive folder ccdrive is my source, and I want to synchronize it to my hard drive E. So I would enter a command like robocopy c:\ccdrive  e:\ccdrive /mir.

There is a complication to this strategy when you extend this process to flash drive. I have a large-capacity flash drive on which I've installed a large number of portable applications and various key backups in part designed to get a backup PC up to speed as quickly as possible in the event of a workhorse failure. Naturally, I want a backup of my flash drive, say F.

So I use a command like the one above to backup F: to e:\fdrive. Here's the counterintuitive thing: You can navigate to e:\fdrive just fine postcopy, but you may not see fdrive in Windows Explorer or other tools. You can fix this in two ways: (1) you can add  /A-:SH preventively to your flashdrive robocopy command, and/or (2) attrib -h -s -a  e:\fdrive. (You can find a related discussion here.)

A Quick Followup on Thunderbird 78 and OpenPGP

 I had not tested OpenPGP capabilities in my last post for a new account and suggested a followup. I'm doing so here.

Actually it's fairly straightforward. I clicked on my second gmail account (without a key pair ). I went to the options menu, clicked on OpenPGP key manager and selected Generate on the popup window. It was easy to submit the new key request. Interestingly, I'm not prompted for a passphrase.  (I am if I want to store a backup copy of my key files.) And then I write a new email choosing Security/Require Encryption to one of my other email accounts, and it's a breeze to read at the other end with the lovely padlock in the upper right.

I do believe that you would have had to install OpenPGP for Windows/Kleopatra. And there are still questions, e.g., why my new key didn't appear in Kleopatra although I seemed to be able to import it into Kleopatra?

Thunderbird 78, Enigmail and Secure Emails

 I migrated to Thunderbird after Microsoft desupported Outlook Express around the mid-2000's. Dealing with large email folders in Windows Mail tested my patience. I also didn't want to upgrade to licensed Outlook. So Thunderbird has been my primary desktop email client during the life of this blog, and it's no accident that multiple posts have touched on Thunderbird.

This week I upgraded to Thunderbird 78; upgrades are always risky since some of your add-ons may not be compatible with the new release. So, for example, a plug-in I was using to access at least a half dozen Google calendars isn't currently available. Of course, I can easily check Google Calendar on my desktop or Android, but it's convenient in my email client if I see, say, a grandniece is celebrating her birthday.

One thing I've looked at doing is improving my email security through PKI technology. Basically there are public/private key pairs that you can use to encrypt and/or establish nonrepudiation of an email source. For example, I can use your public key to encrypt an email so only you can view its content, e.g., by providing a correct passcode/PIN. I can also apply my private key to the email which you could use my public key to verify that I sent said email. (For a related discussion, see here.)

Government (especially military) personnel often use smart tokens/smartcards known as CAC's. (I've discussed CAC's in recent posts.) Basically there are PKI certificates which are paired with your passcode/PIN to work with secure emails, network access and/or endpoint devices, etc.,It's a form of multi-factor authentication: something you have (a token), something you know (the passcode).

In legacy Thunderbird one add-on, Enigmail, has provided an implementation of PKI through integration with OpenPGP (pretty good privacy). I muddled through its implementation. All of this is freeware, no out-of-pocket costs including limited-term certificates, Now I have a large number of email accounts for various purposes, but there are 3 external providers I primarily use (an arbitrary order: hormail/outlook.com, gmail, and yahoo). And so I configured key-pairs for each of the accounts, and tested the functionality among the accounts.

The biggest problem I have with the technology is almost no personal contact or other (business) emails deploy PKI. I use it so infrequently (mostly to check functionality after various upgrades), I'll sometimes have to check one of my password stores to recall my different passphrases for the accounts.

One of the key new features of Thunderbird 78 is native support for OpenPGP, which basically means Enigmail is redundant.  It's fairly straightforward to create a new keypair through OpenPGP Kleopatra, but I haven't come across any tutorials on implementing them in Thunderbird. As time permits, I'll try to add a fourth keypair and perhaps document it in a future post.

One nice thing in Thunderbird for past Enigmail users is they provide a migration option I believe in the options menu. At least the initial steps of the migration were fairly obvious; in my case, in the order yahoo, gmail and outlook.com. What completely threw me off was the fourth prompt, which prompted me for the password for a long randomized alphanumeric string. What the hell? Is it prompting me for some password I forgot to capture in configuring Enigmail a while back?

I noticed there \were 3 such prompts\, so the obvious inference is I had to reenter the same passwords. In what order? I guessed in the earlier migration sequence. Good guess. I'm not sure why the interface was designed that way, but it wasn't obvious.

It's fairly easy to toggle on the signature and/or encryption options (I think through a security menu in the compose window), not to mention adding your public key to the email. And when I opened the email at the target I noticed a nice padlock symbol in the message window.

The New Blogger Interface: Some First Impressions

I don't like being forced into an upgrade, especially where it violates expectations of past experience, makes things less convenient, etc. Back in the 1980's, Coca-Cola decided to change the formula of its classic soft drink to a sweeter version and would not allow the customers to choose their preferred option. Consumers rebelled, hoarding supplies of the legacy formula. To its credit, the company quickly relented, relabeling and producing "Coca Cola Classic". New Coke never did catch on and was eventually  dropped; decades later, the company dropped "classic" on packaging of its legacy formula.

Since starting work on this post, I've discussed some of the issues I have with the new Blogger interface in a segment of my signature political blog here. Ironically, one of my chief complaints, which has to do with Blogger's New Coke approach, doesn't apply to this blog; I do have a link for reverting back to Blogger Classic, although only temporarily, for this blog. I don't know why the older blog doesn't have the link. Many of my issues deal with toggling the compose and html mode. In my daily political "miscellany" post, I'll often include a number of embedded objects, primarily video clips. So typically I'll copy and  paste bits of html code from other sources into html mode. Now the classic mode of html did a beautiful job of maintaining separation of html code and preserving text lines between modes, so, for example, I could effectively insert a couple of lines between a video and the next (existing) section header while inside html mode and those blank lines would carry over to compose mode. There were various functional reasons for inserting blank lines in html mode, including it is an easier way to avoid carryover formatting while in compose mode, e.g., from headline format to normal text format. I could more easily adjust the post format without fiddling with formats in compose mode.

It also makes it easier to find and replace html code. An example is that I've sometimes thought I had copied a video's embedded code into clipboard for insertion (replacing the prior video's code) but the copy failed, and I ended up duplicating the video in the post, which I discovered after publishing the post. Under the classic html mode, it was fairly easy to locate the duplicated video code and replace it for updating the post.

Under the new html format, html code becomes more spaghetti-like in a collapsed format and you need to parse the html ball to make your changes. Spacing in draft mode doesn't map to the compose mode. For instance, my miscellany posts usually include a quote for the day and a daily older music video "interlude" at the bottom. But if the first thing I do after adding a quote is to add my music video of the day, I can separate the quote and video segments by 50 lines in html mode, but if I flip back to display mode, the music video section appears immediately after quote and I have to fuss with compose mode settings to insert intervening post segments. It adds to the busy work of writing and publishing a mixed-mode post. (It isn't as much of an issue in drafting a primarily text post like this one.)

One related aspect I didn't discuss earlier is that Blogger Classic would also provide a way of displaying an embedded video (especially Youtube clips). Now you simply see a gray blob. The (earlier) WYSIWYG compose display didn't work for all but most of the clips I would embed. There is a preview post mode (under both versions) which works to the same desired end. Occasionally I'll run into a clip where I can't see if it works until the post is actually posted. But obviously it was easier for me to verify the clip in a WYSIWYG compose view than to preview or publish the post.

There are other minor points, probably idiosyncratic to my blogging activity. One is the fact that there used to be a checkbox in the blog stats page where you could set a blocking cookie so your own pageviews wouldn't inflate statistics. (Technically, I would prefer that to be true by default without having to constantly check if the cookie is still there.) I'll often tweak a published post for various issues like typos or wording, and maybe up to a half-dozen edits (rare, but it happens) would significantly bias my reader stats. (Some of my blogs have 100 or more pageview posts, but say I probably average less than a dozen on my daily blog;) I have an informal preference to see at least double-digits, but "real" double-digits. I have probably dozens that have capped at 9, but I don't want to cheat just by viewing each post in question. So the point is, if there is a block cookie option in the new Blogger I haven't found it yet. I recall recently I had a delayed browser launch of my published post, and the browser eventually responded with 3 or 4 windows; those all factored into the post statistics.

Finally, there are a number of feature inconsistencies, not that difficult but annoying and not necessarily obvious. An example to make the point: I'll often embed a political cartoon in my miscellany daily post and use the caption function to attribute the artist and the source. I normally had to resize the embedded image under the old format and actually like the initial size under new functionality; if I had to tweak the size, the controls are obvious, while the old controls were more of a toggle switch approach. However, somehow I didn't recognize the caption option at first and ended up manually inserting a caption line in a line following the image the first few times I inserted images and eventually discovered the caption option by playing around with the interface. Maybe the interface was more natural to other people.

It Never Seems To Stop

Technical issues. Hardware, software upgrades, misleading technical information, etc. For example, A USB smartcard reader stops working. After some trial and error, I get it working in another USB port. Was it an issue with the original port, with other connected USB devices, say, an external disk drive, affecting it, a problem with the reader itself, some recent updates, e.g., to device drivers?

As I write, my primary PC is making a second attempt at the latest Windows 10 feature update. It had successfully completed on my newer PC, although it seemed to take forever. I knew something had happened when I took a short nap and woke up to find my PC ready for logon and it did not launch into some sort of  "preparing your desktop" step. I quickly confirmed the update had failed by looking at the Update history. No clue as to what happened, but at least the system had reverted to a usable state. [It has since failed again while I composed this post, at about the 58% completion point. I tried running the Updates troubleshooter to no result. This is not the first time I've run into multiple Update failures.] My older devices are "not yet ready" for the update. I know in the past they eventually get usable updates, but I find it annoying to having different versions across my devices. But even launching Setup/Windows Update can be an issue at times.

Then there's Adobe Flash Player. There are a number of issues here, including its use with certain mandating training software I use. A big upcoming issue is the product's end of life at the end of the year, with all major browsers announcing related end of support. There have been some chronic security lag patching, never mind browsers handle Flash Player support differently, e.g., Chrome vs. Firefox. One of the things I found that using the software through Chrome required manually changing the browser window to enable Flash Player; apparently there's no way to whitelist URLs to allow Flash Player. I discovered the Player issue when I got to a simulation exercise in the courseware and it just wouldn't advance--it was just spinning. I had contacted technical support; he didn't seem to run into the issue using my account--and then at some point he mentioned using Firefox. I recall in the past when using cable TV scheduling with Chrome. I would get some "right-click to enable Adobe" thing but at some point it changed to clicking at the start of the address prompt. However, in the case of the training software, I never got a warning to the effect there was a Flash Player issue.

And then I got to this one course where Flash Player was allowed, but I got a warning the courseware required Flash Player to display the screen. Now what? I remember the support guy had initially tested on Firefox. So I launched my copy of Firefox, only to discover Flash Player wasn't installed, and I eventually realized I had to download the player from Adobe. I remember wondering how to verify the software was installed in Firefox. I eventually launched the courseware and saw something to the effect of right-click to use Flash Player.

I know when I wrote my recent posts on issues with government smartcard CAC's I saw an emphasis in other sources on running the 32-bit version of Internet Explorer in order to use ActiveX functionality in order to use PKI for secure email (digital signature and encryption). If you're running a 64-bit OS, like many of my own PC's have, you should have both 64- and 32-bit versions. Long story short, I started launching the 32-bit iexplore. For other reasons, I started looking at Process Explorer from Sysinternals to look at the iexplore process(es) (right-click/Properties to see the executable path name). That's when I discovered 64-bit IE was still being launched concurrently; more to the point, even if I launched 64-bit IE, I could still use secure email functionality; I think I read somewhere that IE-64 will open a frame to support 32-bit requirements when needed.

On a side note, I have discovered that using a VPN can have some unpleasant side effects. For example, I  found sending an email using Gmail was failing but if I momentarily disconnected the VPN, I could connect to the SMTP server. I've also found at least a dozen secure websites balking they don't recognize my IP. From a security standpoint, I appreciate the checks, but it is a usability tradeoff and somewhat defeats the purpose of using a VPN.

Troubleshooting Secure Email Issues With a CAC

I was tempted to simply add a second addendum to my recent CAC post, but I wanted to expand on relevant comments.

To summarize, a (government) CAC smartcard chip contains PKI infrastructure, including private certificates and associated private keys. If I use third-party software, like ActivClient, I can see 3 certificates (identity, signature, and encryption). The identity certificate is used to authenticate, say, to a government website. The email signature is used to provide non-repudiation of the message source  and encryption ensures confidentiality. In practice, the private infrastructure is protected by the CAC PIN. So, for example, in accessing a secure government website, including email, I'll usually get a pop-up to select the (identity) certificate; there's a second step (on my home system) where it's testing the card reader and reading the CAC;  on an intermittent basis I'll sometimes get a garbage popup saying something like it can't use the certificates on the CAC. What this really means in practice is I need to reseat the CAC--remove the card from the CAC reader and reinsert it. (IMPORTANT practical note: if I'm using Internet Explorer 32-bits,  I need to go to Internet Options, Content Panel and clear the SSL state before I can reread the CAC.) Once I get confirmation that the device is ready and click OK, I should be prompted for my CAC PIN. If successful, I'll usually land at a USG warning banner page (in IE, I'll sometimes have to refresh the webpage). Similar PIN entries occur if I sign and/or encrypt outgoing emails for security.

Now I ran into some weird issues soon after installing/testing S/MIME for use in IE-32 as described in the last message (needed for secure email functionality) I recall being able to pull up a digitally signed and/or encrypted email from another person and sending my own signed/encrypted email. A few days later, I tried pulling up another email from the same person, and the entire message body was blank. What the devil? Was he encrypting from an obsolete public certificate? A separate, weird issue: all of a sudden I couldn't send even regular emails from Outlook Web  Access; clicking on the send button didn't do anything.

I really didn't want to contact the government helpdesk on the issues; I did have a contact with the local group servicing the laptop, and he mentioned he had also run into a blank email issue with OWA but not when using the Outlook client on his laptop (not available to me).

I did a Google search on my issues and found this Microsoft webpage where the user's experience exactly matched mine. When I got to Jeremy Nickels' detailed response  and saw 231 readers had endorsed it, I was convinced I had stumbled across a solution for apparently a common problem. The key steps are up to step 10 and involve a number of tweaks to IE settings, and yes, they resolved my functionality issues. I didn't check militarycac.com; I think when I tested that first signed/encrypted email after installing S/MIME via a related note, I assumed no other tweaks were necessary.