Tuesday, December 22, 2009

FreeOTFE command line

FreeOTFE is a very nice freeware encryption program. It is very similar to the well known TrueCrypt application, but I prefer FreeOTFE since it has a PocketPC version and it's thus more portable.


Like I mentioned in a previous post, I use FreeOTFE in combination with Dropbox. I encrypt a complete portable DropBox folder to make sure all data and account information is kept save whenever I would lose my USB stick. This is possible since FreeOTFE can very easily be used as a portable application and thus perfect for any USB stick.


Since I use my USB stick with portable DropBox and portable Roboform2Go almost every day, I made some small batch scripts to easily start up the FreeOTFE drive and the portable applications within the encrypted volume. So far I have created these scripts:


FreeOTFEAutoLoad.bat: Batch script to mount a volume with a portable FreeOTFE, the volume to mount has to be specified as first parameter when running the script, so for example: FreeOTFEAutoLoad.bat "example.vol"



@echo off
TITLE Auto load FreeOTFE
FreeOTFE\FreeOTFE.exe /portable start /silent
FreeOTFE\FreeOTFE.exe /mount /volume %1%
exit

FreeOTFEStop.bat: Batch script to unmount all volumes that are started using a portable FreeOTFE.



@echo off
TITLE Stop FreeOTFE
:DISMOUNT
FreeOTFE\FreeOTFE.exe /dismount all /silent
IF %ERRORLEVEL% NEQ 0 GOTO DISMOUNTFAILED
GOTO STOPPORTABLE
:STOPPORTABLE
FreeOTFE\FreeOTFE.exe /dismount all /silent /force
FreeOTFE\FreeOTFE.exe /portable stop /silent
GOTO EXIT
:DISMOUNTFAILED
echo Dismount of FreeOTFE failed: %ERRORLEVEL%.
echo Make sure all locks on FreeOTFE disks are removed, then continue this process.
SET INPUT=
set /p INPUT=Press enter to try again, f to force dismount, e to exit.
if /i "%INPUT%" == "f" goto FORCEDISMOUNT
if /i "%INPUT%" == "e" goto EXIT
FreeOTFE\FreeOTFE.exe /dismount all /silent
IF %ERRORLEVEL% NEQ 0 GOTO DISMOUNTFAILED
GOTO STOPPORTABLE
:FORCEDISMOUNT
FreeOTFE\FreeOTFE.exe /dismount all /silent /force
IF %ERRORLEVEL% NEQ 0 GOTO DISMOUNTFAILED
GOTO STOPPORTABLE
:EXIT
exit

FreeOTFE - volume.bat: Batch script to automount a specific volume and launch some script from within the encrypted volume once it is mounted.



@echo on
start "AutoLoad" /wait FreeOTFEAutoLoad.bat "example.vol"

set _Target=NotFound
set _TargetName=REPLACE_THIS_TEXT_WITH_THE_EXACT_NAME_OF_YOUR_MOUNTED_FREEOTFE_VOLUME

for /f usebackq %%a in (`Drives.exe -f %_TargetName%`) do set _Target=%%a

if "%_Target%" == "NotFound" (
echo Unable to find target drive named "%_TargetName%"
goto :EOF
)
pushd %_Target%
%_Target%\SCRIPT_OR_APPLICATION_TO_BE_STARTED_FROM_THE_MOUNTED_VOLUME.bat

When launching FreeOTFE it can take some time to load all cyphers and hashes included, so I created my own archive only including the cypher and hashes I use (the most secure onces). Now FreeOTFE will launch much faster. The application "Drives.exe" is created by Scott Seligman and is used to determine the drive letter based on the name of a volume. The minimal portable FreeOTFE with all scripts above can be downloaded using this link.

Tuesday, November 24, 2009

Extra features for Log4j DailyRollingFileAppender

Log4j has some nice features and supports many appenders, but so far we never had a file appender with all features as it should be. Recently we wanted to write our own appender but before doing so we found Ryan Kimber already had the same idea and did a good job rewriting the original DailyRollingFileAppender: the CustodianDailyRollingFileAppender. On his blog he provides some info on how he updated the appender. We made some very small changes to make it completely working for us and to make sure the log file directory is created if it didn't exist.



  • make sure the directory structure to the specified log file exists

  • create a new log file for each day

  • compress log files older than today

  • remove log files older than a specified number of days


CustodianDailyRollingFileAppender.java source


Example of log4j configuration:



log4j.rootLogger=INFO, FILE
log4j.appender.FILE=com.myt.common.logging.CustodianDailyRollingFileAppender
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%d{MMM dd yyyy HH:mm:ss,SSS} [%t] %-5p %l - %m%n
log4j.appender.FILE.File=/var/log/myt.log
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd
log4j.appender.FILE.MaxNumberOfDays=30
log4j.appender.FILE.CompressBackups=true

Wednesday, November 18, 2009

How to choose the right WiFi channel

If you setup your wireless access point, you may want to configure it in such a way minimizing interference in your neighborhood. I found this interesting post with some tips:



  • There must be a spacing of at least 5 channels (or more) between each WiFi network in order to avoid interferences. Two WiFi networks operating on the same channel are forced to share bandwidth, as they can't "talk" simultaneously, which halves each network's bandwidth. In order to evade this effect, you need to change your access point's channel, but taking the adjacent one won't do it, as WiFi channels are arranged in an overlapping pattern, as you can see in the scheme below. The default channel of most wifi devices is channel 6, so in many cases channel 11 or higher are a good choice. Using NetStumbler one can very easily see which channels is used for each access point.






  • If all your WiFi-devices support 802.11g (the 54 MBit/s WiFi-variant), you should set your router to 802.11g-only mode, as the 802.11b-compatibility impacts on bandwidth and range even among 802.11g-devices.

  • Another possible cause of low performance may be proprietary WiFi acceleration modes like "SuperG", "MAXg", "125 High Speed Mode" or "SpeedBooster", if not all devices in your network support the very same mode, why you should disable those.

  • Also note that a lot of cordless phones in NZ operate at the 2.4 GHz band like Wifi and so most of them cause interferences WiFi, that can't be avoided by a channel change, since those phones use a very broad spectrum or perform permanent frequency hopping.


    If you own a 2.4 GHz phone, try switching it off and removing the power supply of it's base station. In case your wireless signal improves, replace your cordless phone with a new one operating at 1.8 GHz or 5.8 GHz.

  • A cheap and easy solution to extend the coverage of your WiFi environment is to place a repeater at the correct location. It will repeat the wireless signal and extend the coverage (no cables needed), without creating a new network. Many very cheap (+-10euro) wifi access points (for example my DLink DWL-G700AP) can be configured to work as a repeater instead of the default access point functionality.

  • Transmit power: In most cases, the transmit power should be set to the highest value. This maximizes range, which reduces the number of access points and cost of the system. If you're trying to increase the capacity of the network by placing access points closer together, set the power to a lower value to decease overlap and potential interference. Lower power settings also limit the wireless signals from propagating outside the physically controlled area of the facility, which improves security.

  • Service Set IDentifier (SSID): The SSID defines the name of a WLAN that users associate with. By default, the SSID is set to a common value, such as tsunami for Cisco products. In order to improve security, you should change the SSID to a non-default value to minimize unauthorized users from associating with the access point. For even better security, some access points let you disable SSID broadcasting. This keeps most client device operating systems (e.g., Windows XP) from sniffing the SSID from access point beacons and automatically associating with the access point. Someone could, however, obtain the SSID using other sniffing tools that obtain the SSID from 802.11 frames when users first associate with the access point.

  • Data rate: Most access points allow you to identify acceptable data rates. By default, 802.11b access points operate at 1, 2, 5.5, and 11Mbps data rates, depending on the quality of the link between the client device and the access point. As the link quality deteriorates, the access point will automatically throttle down to lower data rates in an attempt to maintain a connection. You can, however, exclude specific data rates. For example, you may want communications only at 11Mbps or not at all. This could be necessary to support higher bandwidth applications.

  • Beacon interval: The beacon interval is the amount of time between access point beacon transmissions. The default value for this interval is generally 10ms, that is 10 beacons sent every second. This is sufficient to support the mobility speed of users within an office environment. You can increase the beacon interval and have lower overhead on the network, but then roaming will likely suffer. It's best to leave this setting alone.

  • Request-to-send / clear-to-send (RTS / CTS): The RTS / CTS function alleviates collisions due to hidden nodes, which is when multiple stations are within range of a common access point but out of range of each other. In most cases, it's best to disable RTS / CTS, but refer to a previous tutorial for cases where RTS / CTS may be beneficial and what threshold values to use.

  • Fragmentation: Fragmentation can help reduce the amount of data needing retransmission when collisions or radio frequency (RF) interference occurs. As with RTS/ CTS, refer to a previous tutorial for cases where fragmentation may be beneficial and applicable threshold values.


Wifi security



  • The impact on the performance by using WEP or WPA really depends on the router. Underpowered old routers don't like the encryption overhead and will slow down somewhat. It is expected to be about 10-15% for either WEP or WPA on older units. In many cases, it's also affected by the speed of the client computer, especially if the WPA encryption is done in driver. Fortunately, this hasn't been the case for many years. These days, there's hardly any slowdown of using WEP or WPA on the performance. However, there's a huge difference in security between WEP and WPA.

  • A nice overview on the weakest to the strongest wireless security capacity is:




    • Considered as not safe:




      • No Security

      • Switching Off SSID: same has No Security. SSID can be easily sniffed even if it is Off

      • MAC Filtering: only to be used if nothing else is available, MAC number can be easily Spoofed

      • WEP64: Easy to "Break" by knowledgeable people

      • WEP128: A little Harder, but still easy to "Break" by knowledgeable people




    • Considered as safe:




      • WPA-PSK: Very Hard to Break

      • WPA-AES: Not functionally Breakable

      • WPA2: Not functionally Breakable







  • If you use Windows XP bellow SP3 and did not updated it, you would have to download the WPA2 patch from Microsoft.

  • The documentation of your Wireless devices (Wireless Router, and Wireless Computer's Card) should state the type of security that is available with your Wireless hardware.

  • All devices MUST be set to the same security level using the same pass phrase. Therefore the security must be set according whatever is the best possible of one of the Wireless devices. I.e. even if most of your system might be capable to be configured to the max. with WPA2, but one device is only capable to be configured to max . of WEP, to whole system must be configured to WEP.

  • Even when using WPA2, one still has to be careful and never use the default WEP or WPA password and default SSID. Different applications exist to recover the default WEP/WPA password based on the SSID. For Alcatel / Thomson SpeedTouch router this online generator can be very easy to recover the default password based on the SSID.

Monday, November 16, 2009

Recovery tools boot USB stick


Based on Hiren's Boot CD 12 I created my personal Boot USB stick to be as complete as possible (containing more than 500 portable tools, 2,30GB). The original Hiren recovery CD contains many very useful tools to recover, tweak or patch pc's, divided into the following categories: Partition Tools, Backup Tools, Recovery Tools, Testing tools, RAM testing tools, Hard disk tools, System information tools, Master Boot Recovery tools, BIOS CMOS tools, Multimedia tools, Password tools, NTFS tools, Browser File manager tools, Other tools, Dos tools, Optimizers, Network tools, Process tools, Registry tools, Startup tools, Tweakers and Antivirus tools. A portable 'mini Windows XP' that can be run from the stick is available as well at boot time.


Many of the tools are available by booting up from the USB stick, but others need to be run into a Windows environment. These windows tools can be easily accessed by using the 'HBCDMenu.exe' tool which will be started when the cd or USB stick is started within a running Windows environment. Since the tools available within the 'HBCDMenu.exe' can be configured very easily using a 'HBCDMenu.csv' file, I created an Excel file to change the configuration an export to the csv file easier. Using this Excel it is much easier to move and rearrange the tools. Next I added all the tools I was still missing to make them available through the 'HBCDMenu.exe' tool. All tools are started using a DOS bat script and an UHARC archive. The archive is extracted in the PC's %temp% folder and started. All tools should be completely portable so no tool settings in the registry are kept after running them. I created a generic batch script to be able to run all tools in different modes: normal, just open a command window, just open an explorer windows, run the tool in Sandboxie, show online info on the tool, convert the tool into a zip file, force extraction of the uharc file. The mode is set by creating a specific file in the %temp% folder.


I keep all my personal files in a secured FreeOTFE file to make sure if I ever lose the stick no personal information can be discovered.


Besides the Hiren tools I also converted the latest BackTrack 4 bootable ISO to make it boot from a USB stick and added this into the Hiren boot screen menu. This live cd linux distribution is focused on penetration testing and perfect for quick and easy WEP cracking.


On the website of Hiren, a good explanation is provided by Hiren on how to easily convert the BootCD into a bootable USB stick using Grub4Dos. I used this 'menu.lst' as boot menu so it includes the launch of the BackTrack live environment and portable Mini Windows XP. Within an Windows environment, this 'autorun.inf' file is used to make it easier to start the 'autorun.exe' tool and other commonly used tools. To keep a backup of all my configuration, I configured a specific portable Dropbox so I can access all my tools online and keep them in sync on different locations.


The USB stick with all the extra tools requires now at least 2,29GB (I use it on a 8GB stick). I also added many of my extra tools into the CD iso file, but to keep it burnable onto a 80minute CD, some of the large tools didn't fit (office portable, tor browser, skype, toad, oracle client).


Compared to the original Hiren 10 boot cd, I've added different Windows tools. In the HBCDMenu cvs creator Excel, the complete list of all the tools are ordered in comprehensive categories. Many of these tools come from Sysinternals and Nirsoft since they provide some very useful portable little tools.


Update 25/11/2009: removed long list of personally added tools
Update 3/12/2010: update for Hiren Boot CD 12

Monday, October 26, 2009

Regular Expressions Regex

I mostly use RegexBuddy to create and test my regular expressions, but I just came across some nice free online tools RegExr , RexV, JavaScript Regex Generator and Nregex to create and test regular expressions. Also useful, a toolbox, RegExp Tools and some examples of some commonly required regular expressions...

Java regex tester

Friday, October 16, 2009

Dropbox - daily usage

Dropbox is a free online service to easily synchronize and backup your data. It can be very useful and it has some very nice features, but it has some limitations too.

Dropbox features:

  • 2GB free online storage, easy registration.
  • Easy sync to different computers / Macs.
  • Files are always reachable using the Dropbox website.
  • Dropbox keeps 30 days version and deleted files history.
  • A public folder with direct static file url's makes it possible to host your site on Dropbox server.
  • Sharing of folders between Dropbox accounts is possible.

Dropbox limitation:

  • No filtering on files / folders is possible.
  • For each account, one folder is synchronized. You need to use Junctions (hardlink NTFS shortcuts (easy Junction explorer extention)) if you want to include external folders in the sync.
  • No default support for multiple account synchronization. You need to use the portable Dropbox (see information below) if you want to synchronize multiple accounts from the same computer at the same time.
  • No option to make Dropbox always request for account password on startup (would be useful for the portable version).

Personal usage scenario's:

  • Synchronization of my AI Roboform data so I have all my login's everywhere. I combine this with the Roboform2go on my USB stick and Dropbox portable to make sure everything is kept in sync.
  • Hosting of files for websites. Instead of having to upload them to different free hosts with ads, I can now use the Dropbox space for easy hosting. Even complete websites are usable when hosted within the 'public' folder of a Dropbox account.
  • Combination of FreeOTFE portable to make sure my USB stick portable Dropbox account is kept save. I place all Dropbox files within a FreeOTFE secured file, since else when losing USB stick, anyone could have access to my Dropbox account by just starting up the portable Dropbox application.
  • Some other tips and tricks for Dropbox usage from LifeHacker. The 'start torrent from anywhere' trick is nice!
  • Since I use my personal SVN I wanted to make a combination of the automatic Dropbox synchronization coupled to the full control SVN synchronization for development projects. By using the Junctions I could link the SVN folders into Dropbox. Now I have an auto sync of files and folders, but I can manually sync with SVN to have an extra backup and history tracking with full control. The downside of this is that all hidden '.svn' folders are kept in sync too within Dropbox and this can take a lot of your Dropbox space. With the selective sync option in Dropbox, you can disable the syncronisation of the .svn folders to save space. But be carefull, when deselecting .svn folders in the Dropbox configuration, it will remove those folders from your local system. So it’s best to first create some dummy empty .svn folders, next disable the sync of these folders and then copy the real .svn folder at the correct location.

Dropbox Portable installation:

DropboxPortableAHK is now available. This makes the use of Dropbox Portable much easier. All download/configuration is now automated and very userfriendly. Just download from the developer website: http://nionsoftware.com/dbpahk/overview

Update 09/01/2011: New version of the Dropbox Portable framework (5.3.4). But this new framework requires a relink! The new version has easier update (just copy official Dropbox setup file in the update folder). Cleanup of blog and added extra info on installation.

Update 20/01/2011: added info from comments to change path in config.db

Update 03/04/2011: link to new DropboPortableAHK version, no more manual tweaks required.

Sunday, October 11, 2009

Keep your batteries in good shape

I red a good article on how to keep your batteries in good shape: Dutch, Babelfish translated English version

Top tips:

  • Make sure to recharge long enough before first usage

  • Always use the original or exactly matching charger

  • Nickel Cadmium (NiCd) batteries should be completely empty before recharging (a battery memory effect will shorten the battery live)
    • These batteries are most often used as AAA or AA rechargeable batteries or older mobile phones

  • Lithium Ion batteries should never be completely empty before recharging (no battery memory, but very sensitive to higher temperatures while charging)
    • These batteries are most often used in mobile phones, pda's and notebooks

  • It can help to put your batteries in a cool environment when you won't use them for some time.

Thursday, October 1, 2009

Folder structure creator - Excel VBS

If you need to create a lot of folders and subfolders, Excel_2007.jpgwith some specific structure, it can be usefull if you can use the power of Excel to make up your folder names and structure. All kind of easy and quick formulas can be used, and once the strucuture is set up, you can easily create the empty folder structure with just one click by using this little VBS macro. The base folder used will depend on the location of the Excel file, so make sure it's saved or copied in the correct folder.
Sub CreateFolderStructure()
'Create folder for all vlues in current sheet
'folders will be created in folder where the excel file was saved
'folders will be created from first row, first column, until empty row is found
'Example expected cell structure: (data starting in current sheet, column A, row 1)
'folder1 subfolder1 subsubfolder1
'folder2
'folder3 subfolder3
'...
'this will result in:
'\folder1\subfolder1\subsubfolder1
'\folder2
'\folder3\subfolder3
'...
Set fs = CreateObject("Scripting.FileSystemObject")
For iRow = 1 To 65000
pathToCreate = ActiveWorkbook.Path
For iColumn = 1 To 65000
currValue = Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, iColumn).Value
If (currValue = "") Then
Exit For
Else
pathToCreate = pathToCreate & "\" & CStr(currValue)
'MsgBox (pathToCreate)
folderToCreate = pathToCreate
If Not (fs.FolderExists(folderToCreate)) Then
fs.CreateFolder (folderToCreate)
End If
End If
Next
Next
End Sub


The Excel sheet with the macro can be downloaded here. Before running the macro make sure the rows and columns of the active sheet are filled in correctly. Next simply run the macro by using the button. foldercreatorexcel1foldercreatorexcel2 foldercreatorexcel3foldercreatorexcel4


If you created to many empty folders by accident, you can easily remove them again using this little tool: Remove Empty Directories


Update 13/11/2009: Modified the Excel VBS script to let you navigate to the desired base folder upon launching the macro, so the Excel file may now be saved at any location, the base folder will have to be specified upon launching the macro.


Update 21/04/2012: Someone commented the VBS code is not working correctly when using some special characters. This is because some characters are not supported by Windows to be used in a file or folder name.


FolderChar


I updated the VBS code in the Excel sheet to make sure these special characters are removed before trying to create the folders.


The Excel sheet is updated, also an Excel template is available.


An example with special characters and the resulting folders created:


FolderSpecialCharsExample


The new VBS code used is:


Sub CreateFolderStructure()
'Create folder for all vlues in current sheet
'folders will be created in folder where the excel file was saved
'folders will be created from first row, first column, until empty row is found
'Example expected cell structure: (data starting in current sheet, column A, row 1)
'folder1    subfolder1  subsubfolder1
'folder2
'folder3    subfolder3
'           subfolder4
'...
'this will result in:
'\folder1\subfolder1\subsubfolder1
'\folder2
'\folder3\subfolder3
'\folder3\subfolder4
'...
    baseFolder = BrowseForFolder
    If (baseFolder = False) Then
        Exit Sub
    End If
    Set fs = CreateObject("Scripting.FileSystemObject")
    For iRow = 2 To 6500
        pathToCreate = baseFolder
        leafFound = False
        For iColumn = 1 To 6500
            currValue = Trim(Replace(Replace(Replace(Replace(Replace(Replace(Replace(Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, iColumn).Value, ":", ""), "*", ""), "?", ""), Chr(34), ""), "<", ""), ">", ""), "|", ""))
            If (currValue = "" And leafFound) Then
                Exit For
            ElseIf (currValue = "") Then
                parentFolder = FindParentFolder(iRow, iColumn)
                parentFolder = Replace(Replace(Replace(Replace(Replace(Replace(Replace(parentFolder, ":", ""), "*", ""), "?", ""), Chr(34), ""), "<", ""), ">", ""), "|", "")
                If (parentFolder = False) Then
                    Exit For
                Else
                    pathToCreate = pathToCreate & "\" & parentFolder
                    If Not (fs.FolderExists(pathToCreate)) Then
                        CreateDirs (pathToCreate)
                    End If
                End If
            Else
                leafFound = True
                pathToCreate = pathToCreate & "\" & currValue
                If Not (fs.FolderExists(pathToCreate)) Then
                    CreateDirs (pathToCreate)
                End If
            End If
        Next
        If (leafFound = False) Then
            Exit For
        End If
    Next
End Sub

Function FindParentFolder(row, column)
    For iRow = row To 0 Step -1
        currValue = Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, column).Value
        If (currValue <> "") Then
            FindParentFolder = CStr(currValue)
            Exit Function
        ElseIf (column <> 1) Then
            leftValue = Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, column - 1).Value
            If (leftValue <> "") Then
                FindParentFolder = False
                Exit Function
            End If
        End If
    Next
End Function


Function BrowseForFolder(Optional OpenAt As Variant) As Variant
     'Function purpose:  To Browser for a user selected folder.
     'If the "OpenAt" path is provided, open the browser at that directory
     'NOTE:  If invalid, it will open at the Desktop level
     
    Dim ShellApp As Object
     
     'Create a file browser window at the default folder
    Set ShellApp = CreateObject("Shell.Application"). _
    BrowseForFolder(0, "Please choose a folder", 0, OpenAt)
     'Set the folder to that selected.  (On error in case cancelled)
    On Error Resume Next
    BrowseForFolder = ShellApp.self.Path
    On Error GoTo 0
     
     'Destroy the Shell Application
    Set ShellApp = Nothing
     
     'Check for invalid or non-entries and send to the Invalid error
     'handler if found
     'Valid selections can begin L: (where L is a letter) or
     '\\ (as in \\servername\sharename.  All others are invalid
    Select Case Mid(BrowseForFolder, 2, 1)
    Case Is = ":"
        If Left(BrowseForFolder, 1) = ":" Then GoTo Invalid
    Case Is = "\"
        If Not Left(BrowseForFolder, 1) = "\" Then GoTo Invalid
    Case Else
        GoTo Invalid
    End Select
     
    Exit Function
     
Invalid:
     'If it was determined that the selection was invalid, set to False
    BrowseForFolder = False
     
End Function

Sub CreateDirs(MyDirName)
' This subroutine creates multiple folders like CMD.EXE's internal MD command.
' By default VBScript can only create one level of folders at a time (blows
' up otherwise!).
'
' Argument:
' MyDirName   [string]   folder(s) to be created, single or
'                        multi level, absolute or relative,
'                        "d:\folder\subfolder" format or UNC
'
' Written by Todd Reeves
' Modified by Rob van der Woude
' http://www.robvanderwoude.com

    Dim arrDirs, i, idxFirst, objFSO, strDir, strDirBuild

    ' Create a file system object
    Set objFSO = CreateObject("Scripting.FileSystemObject")

    ' Convert relative to absolute path
    strDir = objFSO.GetAbsolutePathName(MyDirName)

    ' Split a multi level path in its "components"
    arrDirs = Split(strDir, "\")

    ' Check if the absolute path is UNC or not
    If Left(strDir, 2) = "\\" Then
        strDirBuild = "\\" & arrDirs(2) & "\" & arrDirs(3) & "\"
        idxFirst = 4
    Else
        strDirBuild = arrDirs(0) & "\"
        idxFirst = 1
    End If

    ' Check each (sub)folder and create it if it doesn't exist
    For i = idxFirst To UBound(arrDirs)
        strDirBuild = objFSO.BuildPath(strDirBuild, arrDirs(i))
        If Not objFSO.FolderExists(strDirBuild) Then
            objFSO.CreateFolder strDirBuild
        End If
    Next

    ' Release the file system object
    Set objFSO = Nothing
End Sub


11/10/2012: I updated the VBS code in the Excel sheet to make sure these special characters are removed before trying to create the folders. Extra input information and validation is added to make sure invalid characters can not be used. The Excel sheet is updated


08/08/2013: Updated sheets, added a trim to remove spaces at begin and end of cell value, since this could result in macro exception (see comments)


17/02/2014: I’ve extended this workbook with a new sheet in which the nested folder structure of the filesystem can be imported. Each folder name will be stored in a separate cell respecting the nested structure


The new VBS code is:


  
Sub ImportFolderStructure()
'Import folder structure starting from selected base folder
'each subfolder will be stored in a separete cell
'eg:
'Folder 1|Subfolder1|SubSubfolder1
'Folder 2|Subfolder2
'Folder 3|Subfolder3|SubSubfolder3
'...
    Application.ScreenUpdating = False
    baseFolder = BrowseForFolder
    If (baseFolder = False) Then
        Exit Sub
    End If
    Application.StatusBar = "Folder structure below " & baseFolder & " will be stored in the sheet " & ActiveCell.Worksheet.Name
    StoreSubFolder baseFolder, 1, 0
    Application.StatusBar = "Folder structure below " & baseFolder & " has been stored in the sheet " & ActiveCell.Worksheet.Name
    Range("A2").Select
    Application.ScreenUpdating = True
End Sub

Sub StoreSubFolder(baseFolderObj, ByRef iRow, ByVal iColumn)
    Set fs = CreateObject("Scripting.FileSystemObject")
    Set folderBase = fs.GetFolder(baseFolderObj)
    Set folderBaseSubs = folderBase.SubFolders
    iRow = iRow + 1
    iColumn = iColumn + 1
    For Each subFolder In folderBaseSubs
        Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, iColumn).Value = subFolder.Name
        StoreSubFolder subFolder, iRow, iColumn
    Next
End Sub

Sub ClearImportData()
    Application.ScreenUpdating = False
    Range("A2").Select
    Range(Selection, ActiveCell.SpecialCells(xlLastCell)).Select
    Selection.ClearContents
    Range("A2").Select
    Application.ScreenUpdating = True
End Sub

Sub CreateFolderStructure()
'Create folder for all vlues in current sheet
'folders will be created in folder where the excel file was saved
'folders will be created from first row, first column, until empty row is found
'Example expected cell structure: (data starting in current sheet, column A, row 1)
'folder1    subfolder1  subsubfolder1
'folder2
'folder3    subfolder3
'           subfolder4
'...
'this will result in:
'<currentpath>\folder1\subfolder1\subsubfolder1
'<currentpath>\folder2
'<currentpath>\folder3\subfolder3
'<currentpath>\folder3\subfolder4
'...
    baseFolder = BrowseForFolder
    If (baseFolder = False) Then
        Exit Sub
    End If
    Set fs = CreateObject("Scripting.FileSystemObject")
    For iRow = 2 To 6500
        pathToCreate = baseFolder
        leafFound = False
        For iColumn = 1 To 6500
            currValue = Trim(Replace(Replace(Replace(Replace(Replace(Replace(Replace(Replace(Replace(Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, iColumn).Value, ":", "-"), "*", "-"), "?", "-"), Chr(34), "-"), "<", "-"), ">", "-"), "|", "-"), "/", "-"), "\", "-"))
            Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, iColumn).Value = currValue
            If (currValue = "" And leafFound) Then
                Exit For
            ElseIf (currValue = "") Then
                parentFolder = FindParentFolder(iRow, iColumn)
                If (parentFolder = False) Then
                    Exit For
                Else
                    pathToCreate = pathToCreate & "\" & parentFolder
                    If Not (fs.FolderExists(pathToCreate)) Then
                        CreateDirs (pathToCreate)
                    End If
                End If
            Else
                leafFound = True
                pathToCreate = pathToCreate & "\" & currValue
                If Not (fs.FolderExists(pathToCreate)) Then
                    CreateDirs (pathToCreate)
                End If
            End If
        Next
        If (leafFound = False) Then
            Exit For
        End If
    Next
End Sub

Function FindParentFolder(row, column)
    For iRow = row To 0 Step -1
        currValue = Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, column).Value
        If (currValue <> "") Then
            FindParentFolder = CStr(currValue)
            Exit Function
        ElseIf (column <> 1) Then
            leftValue = Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, column - 1).Value
            If (leftValue <> "") Then
                FindParentFolder = False
                Exit Function
            End If
        End If
    Next
End Function

Function BrowseForFolder(Optional OpenAt As Variant) As Variant
     'Function purpose:  To Browser for a user selected folder.
     'If the "OpenAt" path is provided, open the browser at that directory
     'NOTE:  If invalid, it will open at the Desktop level
     
    Dim ShellApp As Object
     
     'Create a file browser window at the default folder
    Set ShellApp = CreateObject("Shell.Application"). _
    BrowseForFolder(0, "Please choose a folder", 0, OpenAt)
     'Set the folder to that selected.  (On error in case cancelled)
    On Error Resume Next
    BrowseForFolder = ShellApp.self.Path
    On Error GoTo 0
     
     'Destroy the Shell Application
    Set ShellApp = Nothing
     
     'Check for invalid or non-entries and send to the Invalid error
     'handler if found
     'Valid selections can begin L: (where L is a letter) or
     '\\ (as in \\servername\sharename.  All others are invalid
    Select Case Mid(BrowseForFolder, 2, 1)
    Case Is = ":"
        If Left(BrowseForFolder, 1) = ":" Then GoTo Invalid
    Case Is = "\"
        If Not Left(BrowseForFolder, 1) = "\" Then GoTo Invalid
    Case Else
        GoTo Invalid
    End Select
     
    Exit Function
     
Invalid:
     'If it was determined that the selection was invalid, set to False
    BrowseForFolder = False
     
End Function

Sub CreateDirs(MyDirName)
' This subroutine creates multiple folders like CMD.EXE's internal MD command.
' By default VBScript can only create one level of folders at a time (blows
' up otherwise!).
'
' Argument:
' MyDirName   [string]   folder(s) to be created, single or
'                        multi level, absolute or relative,
'                        "d:\folder\subfolder" format or UNC
'
' Written by Todd Reeves
' Modified by Rob van der Woude
' http://www.robvanderwoude.com

    Dim arrDirs, i, idxFirst, objFSO, strDir, strDirBuild
    ' Create a file system object
    Set objFSO = CreateObject("Scripting.FileSystemObject")
    ' Convert relative to absolute path
    strDir = objFSO.GetAbsolutePathName(MyDirName)
    ' Split a multi level path in its "components"
    arrDirs = Split(strDir, "\")
    ' Check if the absolute path is UNC or not
    If Left(strDir, 2) = "\\" Then
        strDirBuild = "\\" & arrDirs(2) & "\" & arrDirs(3) & "\"
        idxFirst = 4
    Else
        strDirBuild = arrDirs(0) & "\"
        idxFirst = 1
    End If

    ' Check each (sub)folder and create it if it doesn't exist
    For i = idxFirst To UBound(arrDirs)
        strDirBuild = objFSO.BuildPath(strDirBuild, arrDirs(i))
        If Not objFSO.FolderExists(strDirBuild) Then
            objFSO.CreateFolder strDirBuild
        End If
    Next

    ' Release the file system object
    Set objFSO = Nothing
End Sub


The Excel file can be downloaded as XLS (template for easy reuse). My Excel sheet with combined macro’s has been updated as well, see this blog post.










Update 27/05/2014: cleanup of the cell types (some cell were saved as type ‘Scientific’ resulting in some strange representation after import of folders named with numbers, Thanks to Jean for reporting)

Wednesday, September 16, 2009

Mouse rocker gestures

Mouse Rocker
Based on the AutoHotKey script from Adam Pash on the Lifehacker site, I made my own version to fit my personal mouse rocker gesture needs.

Basically, a mouse rocker gesture requires that you press one mouse button, hold it down, then press the other. You can rock across the mouse from right-to-left or left-to-right; each direction you rock gives you a different result. Once you get used to this gesture, the name makes perfect sense, and you'll wonder why you weren't mouse rocking your whole life.


I changed the way the script is configured so one can now easily add process names in the rocker.ini file to change the behaviour for a specific application. And I added extra navigation combinations so one should be able to easily make it fit his own needs.
For example in excel you can now easily switch to the next or previous tabbed sheet using mouse rocker gestures. UltraEdit switching between open files is also supported by sending 'Alt + up' or 'Alt + down' keys with the mouse rocker gestures. On first run, it will now also ask if you want to start the tool automatically during windows start. The ini file is created on first run in the directory from where the rocker.exe file is launched. The different groups denote the keys that will be send when a mouse rocker gesture is detected, one can easily add or remove any process name to change the behaviour in a specific application.
[Preferences]
CtrlGroup=itunes.exe
CtrlTabGroup=notepad++.exe,dreamweaver.exe,pidgin.exe
CtrlUpDnGroup=empty
CtrlPgUpDnGroup=excel.exe
AltGroup=firefox.exe,iexplore.exe,opera.exe,feeddemon.exe,explorer.exe
AltUpDnGroup=uedit32.exe
AltPgUpDnGroup=empty
BckspGroup=empty
IgnoreGroup=empty
Startup=1
UpdateCheck=0
Version=0.3



No installation is required, just download the zip file, extract and run the exe. The AutoHotKey source script is included in the zip file.

Update 19/02/2011: links updated

Update 20/12/2017: links updated

Hibernate subquery join using Criteria

hibernate
I recently needed to create max query in hibernate returning an object instead of the maximal value of the field and I wanted to do this using Hibernate Criteria in our JPA environment.
A simple example of what I wanted:
select *
from user
where userid = (select max(userid)
                from user
                where company = 'aCompanyName')

The way to program this in JPA/Hibernate using Criterias, DetachedCriteria and Subqueries. Make sure to use Subqueries.propertyEq instead of Subqueries.eq if you want to join on a field:

public User getMaxUserOfCompany(String companyName) {
Session session = (Session) em.getDelegate();
DetachedCriteria subCriteria = DetachedCriteria.forClass(User.class);
subCriteria.add(Restrictions.eq("company", companyName));
subCriteria.setProjection(Projections.max("userid") );
Criteria criteria = session.createCriteria(User.class);
criteria.add(Subqueries.propertyEq("userid", subCriteria));
return (User) criteria.uniqueResult();
}

Thursday, July 2, 2009

SpeedTouch ADSL router patch

Below is shown how to flash a SpeedTouch router with a new firmware when the administrator password is not known, and then patch the SpeedTouch 716v5 router to disable the ADSL modem and use it as a standard 3-port lan router, 1 port will be used for incoming internet (wan). The configuration was provided by Thomson.



Recently we had to leave Tele2 with theire very nice unlimited offering. We joined Telenet because we coulnd't connect but through a coax cable in our new house.


For Tele2, we had to buy a ADSL modem + router, the Thomson Alcatel SpeedTouch 716v5. It's a very nice router with Wifi and VOIP integrated. Since this router was still working fine, I didn't want to throw it away. But the problem is that Tele2 puts an administrator password on it, and they didn't want to remove the password or provide it to me. To flash a new rom on the router, you also need to have the administrator password.




I was able to find a method to flash a new rom on the router without having the administrator rights. I flashed this (afterwards I used 6.2.29.2) rom onto it:


-connect your computer with a lan cable to the SpeedTouch router


-assign a static ip to your network card, you can just copy the settings you got when you received an IP from the DHCP


-start the flashing by executing the upgrade file


-when the upgrade requests for the router password, use a small screwdriver to push the reset button and keep this reset button pressed. In the meantime power off the SpeedTouch and power it on again. Keep on pressing the reset button and wait for 10 - 15 seconds. The power led will blink red, now you can release the reset button.


-Quickly press back on the upgrade flasher, the software will search again for the SpeedTouch device and continue the upgrade without requesting the administrator password.




I had to do it over some times before all went right, but at the end I managed to flash the default SpeedTouch716v5 rom v 6.1.7.2 and afterwards v6.2.29.2




Once the default rom is installed, you will have Administrator rights by logging in using: username Administrator and emtpy password.




Next, I configured the router to not use ADSL modem anymore, but use the 4th lan port as incoming internet connection and distribute the internet over Wifi and the remaining 3 lan ports with the build in DHCP server. The commands must be put in using telnet: Start -> Run -> cmd -> telnet, login with user Administrator and password (empty by default).


:ppp relay flush
:ppp flush
:eth flush
:atm flush
:atm phonebook flush
:eth bridge ifdelete intf=ethport4
:eth ifadd intf=eth_wan
:eth ifconfig intf=eth_wan dest=ethif4
:eth ifattach intf=eth_wan
:ip ifadd intf=ip_wan_eth dest=eth_wan
:ip ifconfig intf=ip_wan_eth status=up
:ip ifattach intf=ip_wan_eth
:nat ifconfig intf=ip_wan_eth translation=enabled
:dhcp client ifadd intf=ip_wan_eth
:dhcp client ifconfig intf=ip_wan_eth metric=5 dnsmetric=5
:dhcp client rqoptions add intf=ip_wan_eth option=dhcp-lease-time
:dhcp client rqoptions add intf=ip_wan_eth option=dhcp-renewal-time
:dhcp client rqoptions add intf=ip_wan_eth option=dhcp-rebinding-time
:dhcp client rqoptions add intf=ip_wan_eth option=subnet-mask
:dhcp client rqoptions add intf=ip_wan_eth option=classless-static-routes
:dhcp client rqoptions add intf=ip_wan_eth option=default-routers
:dhcp client rqoptions add intf=ip_wan_eth option=classfull-static-routes
:dhcp client rqoptions add intf=ip_wan_eth option=domain-name-servers
:dhcp client ifattach intf=ip_wan_eth
:saveall



You can now connect your incoming internet connection on LAN port 4 and the internet will be distributed over the other LAN ports and Wifi. I tested this method with rom version 6.2. But then the internet was only distributed over the 3 remaining LAN ports, and not over the WIFI connections.
Update 18/04/2010: working with 6.2.29.2.




Finaly, I configured VOIP. I find out Weepee was very cheap and working very well so far. They have a very fast email support. To get a Weepee fixed line telephone number it costs 9 euros. One time fee! You can chose to receive a new number or transfer your existing fixed number, either way it will cost only 9euros. After this one time cost, you only have to pay for the calls you make, and the rates are very competitive.


To configure Weepee VOIP on my SpeedTouch, I used these VOIP settings:



SIP URI: SIP username provided by Weepee (12 numbers, starting with 32)


Username: SIP username provided by Weepee (12 numbers, starting with 32)


Password: SIP password provided by Weepee


Displayname: SIP username provided by Weepee


Abbreviated number: SIP username provided by Weepee


Port: Phone 1



Expert VOIP configuration:



Registrar: SIP server provided by Weepee, ssw7.weepee.org


Registrar Port: 5060


Proxy: SIP server provided by Weepee, ssw7.weepee.org


Proxy prot: 5060


Expire time: 3600





That should do the trick. I hope someone else can benefit from it.


Update: 8/11/2009


WeePee now has native support for Skype over VOIP. You can be called and make calls for free over Skype. To make it work you'll need to follow these steps:


- Login into WeePee client configuration panel: https://ssl.weepee.org/klanten/flash


- Go to 'Information' -> 'Skype'. As mentioned on this page, you need a Skype Business Account. You'll need to create a new account through the Skype Business Account User management panel using the Skype button 'Create a business account'. (It doesn't work by registering an existing account into Skype Business)


- When the Skype business account is created, you'll need to login once using Skype normally. This way, you'll be able to add your contacts into this account too.


- Now you can provided the Skype username and password of the Skype Business Account into WeePee client configuration panel. Click on the 'Create' button to connect Skype into Weepee. Your status should change to 'Online'.


- To make calls over Skype through WeePee, click on the tab menu 'Short numbers' within the WeePee client configuration panel. For each of your Skype contacts, add a shortcode by adding the 'skype#<skypeusername>' for each shortcode. No you can call your Skype contacts by calling for example '25' from your phone.


Update 09/04/2010: new link for speedtouch rom

Update 18/04/2010: new rom version 6.2.29.2 which works with the specific lan configuration

Sunday, May 24, 2009

Cobol data + Cobol copybook + Java conversion

Although I'm absolutely not a fan of Cobol, it's still inevitable in the Finance IT development sector.

Lately, we needed a way to let Cobol and Java data work together, and we didn't want to hard code the complete data structure. To make this possible we noticed different non-free applications exist, but the open source project Cb2Xml got our attention.

This project already worked out some Java code to parse a Cobol copybook and convert it into an XML representation. (The copybook can be seen as the interface of the Cobol data). But this project was mend to import and export data from xml to Cobol stream and vice-versa, while we needed some Java objects to work with the received Cobol data input.

So I added some extra code to convert a Cobol data stream (String) into a Java object (using a Hashtable internally). Now, it's possible to provide a String of Cobol data and its interface definition (Cobol copybook file) and return a Java CobolElements object. One can search in this CobolElements object based on the name or xpath. I made a simple example to test, which might make it clear how to use the code.

A simple example of Cobol copybook used as data interface:

 01 ACCOUNT-GROUP                                               .
  02 ACCOUNT                                              .
   03 DFND                                 PIC  9(4)             .
   03 DCTB                                                       .
    04 DUPD                                PIC  9(8)             .
    04 DNUMCTB                                                   .
     05 DCTB-12                            PIC  9(12)            .
     05 DCTB-04                            PIC  9(4)             .
     05 DFMT                               PIC  9(2)             .
    04 DCDRCTB                                                   .
     05 DRGOCDRCTB                         PIC  9(1)             .
     05 DROFCMCCDRCTB                      PIC  9(3)             .
     05 DBRACMCCDRCTB                                            .
      06 DBRACMCCDRCTB                     PIC  9(6)             .
      06 GBRACDRCTB                                               
             REDEFINES DBRACMCCDRCTB                             .
       07 DROFCDRCTB                       PIC  9(3)             .
       07 DBRACDRCTB                       PIC  9(3)             .
    04 DBLK                                PIC  9(2)              
             OCCURS 5                                            .
    04 DBLK2                               PIC  9(3)              
             OCCURS 5                                            .


The data that should match this copybook:



input = "00000000000039300022955600000123703602231122334455111222333444555"; 


Converting the data to a Java CobolElements object:




  • Converting the copybook to an xml representation (this XML Document should be created once for each copybook and can be cached):



Document cb2doc = Cb2Xml.convert(new File(_cobolCopybookFileName), _debug); 



  • Converting the Cobol data stream (input string) to it's matching Java representation:



CobolElements cobolElements = Dat2Java.convertWithDoc(input, cb2doc); 


Retrieving data from the CobolElements object:



CobolElement childElement = cobolElements.retrieveChildElement("GOUTCTT-CSISEQ/GANSCTT-CSISEQ/GCTB/GNUMCTB/NCTB-12"); 
System.out.println(childElement.getData()); 
//returns: 393000229556 
childElement = cobolElements.retrieveChildElement("ACCOUNT-GROUP/ACCOUNT/DCTB/DBLK[3]"); 
System.out.println(childElement.getData()); 
//returns: 44 


Download source and jar (zip 1,78MB) (link updated 16/09/2009)


Update (27/11/2009): After creating the cobol2java, we found another interesting open source project called LegStar. They have a completely worked out solution, while the code we use is quite basic and only usable with simple copy books...

Eclipse project

eclipse Each project (workspace) in Eclipse has it's own .project file. To easily open the correct workspace with Eclipse, I wrote a little batch script that can be associated in Windows with the .project files. Now, I only have to double click the .project file and Eclipse will be loaded in the correct workspace.

  • Save the batch script bellow and make sure the correct Eclipse.exe file is used. You might want to change the Eclipse options if required.
  • Double click on a .project file, the first time Windows will ask how to open this file. Choose the .bat batch file script to open your .project file and make sure to make Windows remembers this option.
  • Eclipse will now open with the correct workspace loaded.
@echo off
set ECLIPSE_BIN=R:\tools\eclipse\eclipse.exe
set ECLIPSE_OPTIONS=-refresh -showlocation -Xmx512M -XX:MaxPermSize=512m
set PROJECT_FULL_PATH=%1%
set PROJECT_FOLDER=%PROJECT_FULL_PATH:.project=%
cd %PROJECT_FOLDER%
cd ..
set PROJECT_WORKSPACE="%CD%"
start "eclipse" "%ECLIPSE_BIN%" %ECLIPSE_OPTIONS% -data %PROJECT_WORKSPACE%
@echo on


Download batch script.

Monday, May 11, 2009

LogExpert - Windows tail freeware

Today I tested yet another Windows tail application, LogExpert, and I was very amazed with its features!
It has most of the features I'm used to work with in BareTail Pro, like adding some filters (regex) or highlighting lines matching some text. And unlike WinLogTail, it does support opening multiple files at one with a tabbed interface as in BareTail.
Besides all this, I really appreciate the following functions:
  • run some specified application and providing some information of the selected line. This way, one can easily open the tailed file within UltraEdit on the selected line.
  • parse the timestamps of a log file and show these in separated columns
  • freeze some columns to make these alway be shown and only scrolling other columns (horizontal scrolling)
  • advanced filtering with possibility to show some lines before or after the line matching your filter
  • activate an 'edit' mode to easily copy some parts of a line
  • add some bookmarks and comments on a line to easily find these back afterwards
  • support for multifiles (show files as app.log, app.log1, app.log2, etc as one big file)
  • support for different files provided on the command line while opening, so the scripts we used to open many files with BareTail also work with LogExpert (but opening many files from shared drives is not as lighting fast as with BareTail)

So, compared to BareTail it has many benefits and it's completely free! But one should know, it's not always as fast and stable as BareTail, but so far it worked very well for me.

Some nice screenshots (but those only show a small subset of all it's features):

Sunday, April 26, 2009

Find-Replace text in files from command line

In Linux one can easily find and replace some text in files by using the sed application. In Windows, I couldn’t find a decent build in equivalent. I ended by using the ported version of sed for Windows. I works quite well using the same syntax and doesn’t require to be installed. Just make sure sed.exe, libiconv2.dll and libintl3.dll are existing in the same folder or on the classpath.

One should know any file altered by sed for Windows will result in this file being also converted from Windows line ending (using 2 characters: Carriage Return + Line Feed) to Linux line endings (only 1 character: Line Feed). This can sometime give some strange results. To overcome this, one should make sure the last replacement in the file is set to replace the Line Feeds back to Carriage Return + Line Feed:

sed -i "s/$/\r/" "%DIR%\file.txt"


Some other useful sed commands are available from sed sourcefourge:



FILE SPACING:
 # double space a file
 sed G
 # double space a file which already has blank lines in it. Output file
 # should contain no more than one blank line between lines of text.
 sed '/^$/d;G'
 # triple space a file
 sed 'G;G'
 # undo double-spacing (assumes even-numbered lines are always blank)
 sed 'n;d'
 # insert a blank line above every line which matches "regex"
 sed '/regex/{x;p;x;}'
 # insert a blank line below every line which matches "regex"
 sed '/regex/G'
 # insert a blank line above and below every line which matches "regex"
 sed '/regex/{x;p;x;G;}'
NUMBERING:
 # number each line of a file (simple left alignment). Using a tab (see
 # note on '\t' at end of file) instead of space will preserve margins.
 sed = filename | sed 'N;s/\n/\t/'
 # number each line of a file (number on left, right-aligned)
 sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'
 # number each line of file, but only print numbers if line is not blank
 sed '/./=' filename | sed '/./N; s/\n/ /'
 # count lines (emulates "wc -l")
 sed -n '$='
TEXT CONVERSION AND SUBSTITUTION:
 # IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # works on ssed, gsed 3.02.80 or higher
 # IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$/`echo -e \\\r`/"            # command line under ksh
 sed 's/$'"/`echo \\\r`/"             # command line under bash
 sed "s/$/`echo \\\r`/"               # command line under zsh
 sed 's/$/\r/'                        # gsed 3.02.80 or higher
 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$//"                          # method 1
 sed -n p                             # method 2
 # IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 # Can only be done with UnxUtils sed, version 4.0.7 or higher. The
 # UnxUtils version can be identified by the custom "--text" switch
 # which appears when you use the "--help" switch. Otherwise, changing
 # DOS newlines to Unix newlines cannot be done with sed in a DOS
 # environment. Use "tr" instead.
 sed "s/\r//" infile >outfile         # UnxUtils sed v4.0.7 or higher
 tr -d \r <infile >outfile            # GNU tr version 1.22 or higher
 # delete leading whitespace (spaces, tabs) from front of each line
 # aligns all text flush left
 sed 's/^[ \t]*//'                    # see note on '\t' at end of file
 # delete trailing whitespace (spaces, tabs) from end of each line
 sed 's/[ \t]*$//'                    # see note on '\t' at end of file
 # delete BOTH leading and trailing whitespace from each line
 sed 's/^[ \t]*//;s/[ \t]*$//'
 # insert 5 blank spaces at beginning of each line (make page offset)
 sed 's/^/     /'
 # align all text flush right on a 79-column width
 sed -e :a -e 's/^.\{1,78\}$/ &/;ta'  # set at 78 plus 1 space
 # center all text in the middle of 79-column width. In method 1,
 # spaces at the beginning of the line are significant, and trailing
 # spaces are appended at the end of the line. In method 2, spaces at
 # the beginning of the line are discarded in centering the line, and
 # no trailing spaces appear at the end of lines.
 sed  -e :a -e 's/^.\{1,77\}$/ & /;ta'                     # method 1
 sed  -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/'  # method 2
 # substitute (find and replace) "foo" with "bar" on each line
 sed 's/foo/bar/'             # replaces only 1st instance in a line
 sed 's/foo/bar/4'            # replaces only 4th instance in a line
 sed 's/foo/bar/g'            # replaces ALL instances in a line
 sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
 sed 's/\(.*\)foo/\1bar/'            # replace only the last case
 # substitute "foo" with "bar" ONLY for lines which contain "baz"
 sed '/baz/s/foo/bar/g'
 # substitute "foo" with "bar" EXCEPT for lines which contain "baz"
 sed '/baz/!s/foo/bar/g'
 # change "scarlet" or "ruby" or "puce" to "red"
 sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'   # most seds
 gsed 's/scarlet\|ruby\|puce/red/g'                # GNU sed only
 # reverse order of lines (emulates "tac")
 # bug/feature in HHsed v1.5 causes blank lines to be deleted
 sed '1!G;h;$!d'               # method 1
 sed -n '1!G;h;$p'             # method 2
 # reverse each character on the line (emulates "rev")
 sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
 # join pairs of lines side-by-side (like "paste")
 sed '$!N;s/\n/ /'
 # if a line ends with a backslash, append the next line to it
 sed -e :a -e '/\\$/N; s/\\\n//; ta'
 # if a line begins with an equal sign, append it to the previous line
 # and replace the "=" with a single space
 sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
 # add commas to numeric strings, changing "1234567" to "1,234,567"
 gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta'                     # GNU sed
 sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta'  # other seds
 # add commas to numbers with decimal points and minus signs (GNU sed)
 gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'
 # add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
 gsed '0~5G'                  # GNU sed only
 sed 'n;n;n;n;G;'             # other seds
SELECTIVE PRINTING OF CERTAIN LINES:
 # print first 10 lines of file (emulates behavior of "head")
 sed 10q
 # print first line of file (emulates "head -1")
 sed q
 # print the last 10 lines of a file (emulates "tail")
 sed -e :a -e '$q;N;11,$D;ba'
 # print the last 2 lines of a file (emulates "tail -2")
 sed '$!N;$!D'
 # print the last line of a file (emulates "tail -1")
 sed '$!d'                    # method 1
 sed -n '$p'                  # method 2
 # print the next-to-the-last line of a file
 sed -e '$!{h;d;}' -e x              # for 1-line files, print blank line
 sed -e '1{$q;}' -e '$!{h;d;}' -e x  # for 1-line files, print the line
 sed -e '1{$d;}' -e '$!{h;d;}' -e x  # for 1-line files, print nothing
 # print only lines which match regular expression (emulates "grep")
 sed -n '/regexp/p'           # method 1
 sed '/regexp/!d'             # method 2
 # print only lines which do NOT match regexp (emulates "grep -v")
 sed -n '/regexp/!p'          # method 1, corresponds to above
 sed '/regexp/d'              # method 2, simpler syntax
 # print the line immediately before a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{g;1!p;};h'
 # print the line immediately after a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{n;p;}'
 # print 1 line of context before and after regexp, with line number
 # indicating where the regexp occurred (similar to "grep -A1 -B1")
 sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
 # grep for AAA and BBB and CCC (in any order)
 sed '/AAA/!d; /BBB/!d; /CCC/!d'
 # grep for AAA and BBB and CCC (in that order)
 sed '/AAA.*BBB.*CCC/!d'
 # grep for AAA or BBB or CCC (emulates "egrep")
 sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d    # most seds
 gsed '/AAA\|BBB\|CCC/!d'                        # GNU sed only
 # print paragraph if it contains AAA (blank lines separate paragraphs)
 # HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
 # print paragraph if it contains AAA and BBB and CCC (in any order)
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
 # print paragraph if it contains AAA or BBB or CCC
 sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
 gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d'         # GNU sed only
 # print only lines of 65 characters or longer
 sed -n '/^.\{65\}/p'
 # print only lines of less than 65 characters
 sed -n '/^.\{65\}/!p'        # method 1, corresponds to above
 sed '/^.\{65\}/d'            # method 2, simpler syntax
 # print section of file from regular expression to end of file
 sed -n '/regexp/,$p'
 # print section of file based on line numbers (lines 8-12, inclusive)
 sed -n '8,12p'               # method 1
 sed '8,12!d'                 # method 2
 # print line number 52
 sed -n '52p'                 # method 1
 sed '52!d'                   # method 2
 sed '52q;d'                  # method 3, efficient on large files
 # beginning at line 3, print every 7th line
 gsed -n '3~7p'               # GNU sed only
 sed -n '3,${p;n;n;n;n;n;n;}' # other seds
 # print section of file between two regular expressions (inclusive)
 sed -n '/Iowa/,/Montana/p'             # case sensitive
SELECTIVE DELETION OF CERTAIN LINES:
 # print all of file EXCEPT section between 2 regular expressions
 sed '/Iowa/,/Montana/d'
 # delete duplicate, consecutive lines from a file (emulates "uniq").
 # First line in a set of duplicate lines is kept, rest are deleted.
 sed '$!N; /^\(.*\)\n\1$/!P; D'
 # delete duplicate, nonconsecutive lines from a file. Beware not to
 # overflow the buffer size of the hold space, or else use GNU sed.
 sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
 # delete all lines except duplicate lines (emulates "uniq -d").
 sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
 # delete the first 10 lines of a file
 sed '1,10d'
 # delete the last line of a file
 sed '$d'
 # delete the last 2 lines of a file
 sed 'N;$!P;$!D;$d'
 # delete the last 10 lines of a file
 sed -e :a -e '$d;N;2,10ba' -e 'P;D'   # method 1
 sed -n -e :a -e '1,10!{P;N;D;};N;ba'  # method 2
 # delete every 8th line
 gsed '0~8d'                           # GNU sed only
 sed 'n;n;n;n;n;n;n;d;'                # other seds
 # delete lines matching pattern
 sed '/pattern/d'
 # delete ALL blank lines from a file (same as "grep '.' ")
 sed '/^$/d'                           # method 1
 sed '/./!d'                           # method 2
 # delete all CONSECUTIVE blank lines from file except the first; also
 # deletes all blank lines from top and end of file (emulates "cat -s")
 sed '/./,/^$/!d'          # method 1, allows 0 blanks at top, 1 at EOF
 sed '/^$/N;/\n$/D'        # method 2, allows 1 blank at top, 0 at EOF
 # delete all CONSECUTIVE blank lines from file except the first 2:
 sed '/^$/N;/\n$/N;//D'
 # delete all leading blank lines at top of file
 sed '/./,$!d'
 # delete all trailing blank lines at end of file
 sed -e :a -e '/^\n*$/{$d;N;ba' -e '}'  # works on all seds
 sed -e :a -e '/^\n*$/N;/\n$/ba'        # ditto, except for gsed 3.02.*
 # delete the last line of each paragraph
 sed -n '/^$/{p;h;};/./{x;/./p;}'
SPECIAL APPLICATIONS:
 # remove nroff overstrikes (char, backspace) from man pages. The 'echo'
 # command may need an -e switch if you use Unix System V or bash shell.
 sed "s/.`echo \\\b`//g"    # double quotes required for Unix environment
 sed 's/.^H//g'             # in bash/tcsh, press Ctrl-V and then Ctrl-H
 sed 's/.\x08//g'           # hex expression for sed 1.5, GNU sed, ssed
 # get Usenet/e-mail message header
 sed '/^$/q'                # deletes everything after first blank line
 # get Usenet/e-mail message body
 sed '1,/^$/d'              # deletes everything up to first blank line
 # get Subject header, but remove initial "Subject: " portion
 sed '/^Subject: */!d; s///;q'
 # get return address header
 sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
 # parse out the address proper. Pulls out the e-mail address by itself
 # from the 1-line return address header (see preceding script)
 sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
 # add a leading angle bracket and space to each line (quote a message)
 sed 's/^/> /'
 # delete leading angle bracket & space from each line (unquote a message)
 sed 's/^> //'
 # remove most HTML tags (accommodates multiple-line tags)
 sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
 # extract multi-part uuencoded binaries, removing extraneous header
 # info, so that only the uuencoded portion remains. Files passed to
 # sed must be passed in the proper order. Version 1 can be entered
 # from the command line; version 2 can be made into an executable
 # Unix shell script. (Modified from a script by Rahul Dhesi.)
 sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode   # vers. 1
 sed '/^end/,/^begin/d' "$@" | uudecode                    # vers. 2
 # sort paragraphs of file alphabetically. Paragraphs are separated by blank
 # lines. GNU sed uses \v for vertical tab, or any unique char will do.
 sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
 gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'
 # zip up each .TXT file individually, deleting the source file and
 # setting the name of each .ZIP file to the basename of the .TXT file
 # (under DOS: the "dir /b" switch returns bare filenames in all caps).
 echo @echo off >zipup.bat
 dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat