Tuesday, November 24, 2009

Extra features for Log4j DailyRollingFileAppender

Log4j has some nice features and supports many appenders, but so far we never had a file appender with all features as it should be. Recently we wanted to write our own appender but before doing so we found Ryan Kimber already had the same idea and did a good job rewriting the original DailyRollingFileAppender: the CustodianDailyRollingFileAppender. On his blog he provides some info on how he updated the appender. We made some very small changes to make it completely working for us and to make sure the log file directory is created if it didn't exist.

  • make sure the directory structure to the specified log file exists
  • create a new log file for each day
  • compress log files older than today
  • remove log files older than a specified number of days

CustodianDailyRollingFileAppender.java source

Example of log4j configuration:


log4j.rootLogger=INFO, FILE

log4j.appender.FILE=com.myt.common.logging.CustodianDailyRollingFileAppender
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%d{MMM dd yyyy HH:mm:ss,SSS} [%t] %-5p %l - %m%n
log4j.appender.FILE.File=/var/log/myt.log
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd
log4j.appender.FILE.MaxNumberOfDays=30
log4j.appender.FILE.CompressBackups=true

Wednesday, November 18, 2009

How to chose the right WiFi channel

If you setup your wireless access point, you may want to configure it in such a way minimizing interference in your neighbourhood. I found this interesting post with some tips:

  • There must be a spacing of at least 5 channels (or more) between each WiFi network in order to avoid interferences. Two WiFi networks operating on the same channel are forced to share bandwidth, as they can't "talk" simultaneously, which halves each network's bandwidth. In order to evade this effect, you need to change your access point's channel, but taking the adjacent one won't do it, as WiFi channels are arranged in an overlapping pattern, as you can see in the scheme below. The default channel of most wifi devices is channel 6, so in many cases channel 11 or higher are a good choice. Using NetStumbler one can very easily see which channels is used for each access point.

  • If all your WiFi-devices support 802.11g (the 54 MBit/s WiFi-variant), you should set your router to 802.11g-only mode, as the 802.11b-compatibility impacts on bandwidth and range even among 802.11g-devices.
  • Another possible cause of low performance may be proprietary WiFi acceleration modes like "SuperG", "MAXg", "125 High Speed Mode" or "SpeedBooster", if not all devices in your network support the very same mode, why you should disable those.
  • Also note that a lot of cordless phones in NZ operate at the 2.4 GHz band like Wifi and so most of them cause interferences WiFi, that can't be avoided by a channel change, since those phones use a very broad spectrum or perform permanent frequency hopping.
    If you own a 2.4 GHz phone, try switching it off and removing the power supply of it's base station. In case your wireless signal improves, replace your cordless phone with a new one operating at 1.8 GHz or 5.8 GHz.

Wifi security

  • The impact on the performance by using WEP or WPA really depends on the router. Underpowered old routers don't like the encryption overhead and will slow down somewhat. It is expected to be about 10-15% for either WEP or WPA on older units. In many cases, it's also affected by the speed of the client computer, especially if the WPA encryption is done in driver. Fortunately, this hasn't been the case for many years. These days, there's hardly any slowdown of using WEP or WPA on the performance. However, there's a huge difference in security between WEP and WPA.
  • A nice overview on the weakest to the strongest wireless security capacity is:
    • Considered as not safe:
      • No Security
      • Switching Off SSID: same has No Security. SSID can be easily sniffed even if it is Off
      • MAC Filtering: only to be used if nothing else is available, MAC number can be easily Spoofed
      • WEP64: Easy to "Break" by knowledgeable people
      • WEP128: A little Harder, but still easy to "Break" by knowledgeable people
    • Considered as safe:
      • WPA-PSK: Very Hard to Break
      • WPA-AES: Not functionally Breakable
      • WPA2: Not functionally Breakable
  • If you use Windows XP bellow SP3 and did not updated it, you would have to download the WPA2 patch from Microsoft.
  • The documentation of your Wireless devices (Wireless Router, and Wireless Computer's Card) should state the type of security that is available with your Wireless hardware.
  • All devices MUST be set to the same security level using the same pass phrase. Therefore the security must be set according whatever is the best possible of one of the Wireless devices. I.e. even if most of your system might be capable to be configured to the max. with WPA2, but one device is only capable to be configured to max . of WEP, to whole system must be configured to WEP.
  • Even when using WPA2, one still has to be careful and never use the default WEP or WPA password and default SSID. Different applications exist to recover the default WEP/WPA password based on the SSID. For Alcatel / Thomson SpeedTouch router this online generator can be very easy to recover the default password based on the SSID.

Monday, November 16, 2009

Recovery tools boot USB stick

Based on Hiren's Boot CD 10 I created my personal Boot USB stick to be as complete as possible (containing more than 500 portable tools, 2,30GB). The original Hiren recovery CD contains many very useful tools to recover, tweak or patch pc's, divided into the following categories: Partition Tools, Backup Tools, Recovery Tools, Testing tools, RAM testing tools, Hard disk tools, System information tools, Master Boot Recoverd tools, BIOS CMOS tools, Multimedia tools, Password tools, NTFS tools, Browser File manager tools, Other tools, Dos tools, Optimizers, Network tools, Process tools, Registry tools, Startup tools, Tweakers and Antivirus tools. A portable 'mini Windows XP' that can be run from the stick is available as well at boot time.

Many of the tools are available by booting up from the USB stick, but others need to be run into a Windows environment. These windows tools can be easily accessed by using the 'Autorun.exe' tool which will be started when the cd or USB stick is started within a running Windows environment. Since the tools available within the 'Autorun.exe' can be configured very easily using a 'Autorun.csv' file, I created an Excel file to change the confiugration an export to the csv file easier. Using this Excel it is much easier to move and rearrange the tools. Next I added all the tools I was still missing to make them available through the 'Autorun.exe' tool. All tools are started using a DOS bat script and an UHARC archive. The archive is extracted in the PC's %temp% folder and started. All tools should be completely portable so no tool settings in the registry are kept after running them.

I keep all my personal files in a secured FreeOTFE file to make sure if I ever lose the stick no personal information can be discovered.

Besides the Hiren tools I also converted the latest BackTrack 4 bootable ISO to make it boot from a USB stick and added this into the Hiren boot screen menu. This live cd linux distribution is focused on penetration testing and perfect for quick and easy WEP cracking.

On the website of Hiren, a good explanation is provided by Hiren on how to easily convert the BootCD into a bootable USB stick using Grub4Dos. I used this 'menu.lst' as boot menu so it includes the launch of the BackTrack live environment and portable Mini Windows XP. Within an Windows environment, this 'autorun.inf' file is used to make it easier to start the 'autorun.exe' tool and other commonly used tools. To keep a backup of all my confiugration, I configured a specific portable Dropbox so I can access all my tools online and keep them in sync on different locations.

The USB stick with all the extra tools requires now at least 2,29GB (I use it on a 8GB stick). I also added many of my extra tools into the CD iso file, but to keep it burnable onto a 80minute CD, some of the large tools didn't fit (office portable, tor browser, skype, toad, oracle client).

Compared to the original Hiren 10 boot cd, I've added the following Windows tools. These are NOT available within the original Hiren boot cd. In the autorun cvs creator Excel, the complete list of all the tools are ordered in compehensive categories. Many of these tools come from Sysinternals and Nirsoft since they provide some very useful portable little tools.

  • AccessChk
  • AccessEnum
  • AdAware
  • ADExplorer
  • ADInsight
  • ADRestore
  • AdvancedIPScanner
  • AdvancedLANScanner
  • AdvancedPDFRepair
  • AdvancedPortScanner
  • AlternateStreamView
  • AnyReader
  • AutoLogon
  • Autorun.xls
  • awatch
  • BackTrack
  • BackupKeyRecovery
  • BareGrep
  • BareTail Pro
  • BeyondCompare
  • BGInfo
  • Bios
  • BiosAgent
  • BiosInformationTool
  • BIOSPatcher
  • BitChe
  • bluescreenview
  • bluetoothview
  • BurnCdCC
  • CacheSet
  • CainAbel
  • ChromeCacheView
  • ChromePass
  • ciadv
  • ClamWin
  • CleanAfterMe
  • CleanDumpTempExit
  • CleanTemp
  • CleanTempExit
  • clipboardic
  • CmosPwd
  • COFEE
  • compmgmt
  • conadvpass
  • Contig
  • Convert
  • CoreInfo
  • cprocess
  • CreateFolderStructure
  • Dbgview
  • DellBiosBoot
  • Desktops
  • deviceioview
  • DevManView
  • devmgmt
  • dfrg
  • dialupass
  • Disk2VHD
  • DiskExt
  • DiskImager
  • diskmgmt
  • DiskMon
  • DiskUsage
  • DiskView
  • dllexp
  • dnsdataview
  • dotnetresourcesextract
  • downtester
  • driverview
  • DropBox
  • DumpSEC
  • eventvwr
  • ExactAudioCopy
  • Exit
  • ExploreTemp
  • ExpressBurn
  • fastresolver
  • faview
  • FileSieve
  • files
  • filetypesman
  • FireFox
  • FlashBoot
  • FormatFactory
  • FormFiller
  • FoxitReader
  • FreeOTFE
  • FreeOTFEExplorer
  • FreeOTFEFull
  • fsmgmt
  • gdiview
  • GenuineAdvantageValidation
  • Gimp
  • GMailPasswordRecovery
  • GoogleTranslateClient
  • gpedit
  • Grub4DOS
  • HamachiBuri
  • Handle
  • hashmyfiles
  • heapmemview
  • Hex2Dec
  • HideHiddenFiles
  • htmlastext
  • iconsext
  • iecacheview
  • iecv
  • iedesignmode
  • iehv
  • iepv
  • insideclipboard
  • installedcodec
  • ipnetinfo
  • IrfanView
  • JavaDecompilerGUI
  • Junction
  • KMPlayer
  • LDMDump
  • LeakTest
  • LessMSIerables
  • LimeWire
  • ListDLLs
  • livecontactsview
  • LiveKD
  • LoadOrder
  • LogExpert
  • LogonSessions
  • lsasecretsdump
  • lsasecretsview
  • lusrmgr
  • macaddressview
  • MagicBios
  • McAfee
  • MirandaPortable
  • monitorinfoview
  • MouseRocker
  • MoveFile
  • mozillacacheview
  • mozillahistoryview
  • muicacheview
  • myeventviewer
  • mylastsearch
  • myuninst
  • mzcv
  • netpass
  • netresview
  • NetStumbler
  • NetworkSecurityAuditor
  • NiBiTor
  • NirCmd
  • nk2view
  • Notepad++
  • ntmsmgr
  • ntmsoprq
  • OB1
  • office-access
  • office-excel
  • office-ois
  • office-powerpoint
  • office-word
  • officeins
  • openwithview
  • operacacheview
  • OracleInstantClient
  • outlookattachview
  • outlookstatview
  • OverDisk
  • Paint.NET
  • PartitionMagic
  • PasswareKit
  • passwordfox
  • PathTooLong
  • PendMove
  • perfmon
  • PhoenixBiosEditor
  • PhotoResizer
  • pinginfoview
  • PipeList
  • Pixie
  • PortMon
  • PowerCalc
  • PowerPacket
  • ProcDump
  • ProcMon
  • ProductKeyExplorer
  • PsExec
  • PsFile
  • PsGetSid
  • PsInfo
  • PsKill
  • PsList
  • PsLoggedOn
  • PsLogList
  • PsPasswd
  • PsService
  • PsShutdown
  • PsSuspend
  • pstpassword
  • PuTTY
  • recentfilesview
  • RED
  • RegDelNull
  • regdllview
  • RegexBuddy
  • regfromapp
  • ResHacker
  • resourcesextract
  • RoboForm2Go
  • runasdate
  • SafeEmailLinkCreator
  • SDelete
  • searchmyfiles
  • secpol
  • SecurAble
  • seqdownload
  • ServerIdentification
  • services
  • serviwin
  • ShareEnum
  • shellbagsview
  • ShellRunAs
  • shmnview
  • ShowHiddenFiles
  • ShutdownTimer
  • SigCheck
  • SimCardDataRecovery
  • siteshoter
  • Skype
  • smsniff
  • sniffpass
  • socketsniff
  • specialfoldersview
  • SplashID
  • SpotAuditor
  • Strings
  • Sync
  • sysexp
  • SysinternalsSuite
  • TagRename
  • TempToUSB
  • TightVNC-server
  • TightVNC-viewer
  • Toad
  • TorBrowser
  • tortoisesvn
  • UltraEdit
  • UltraISO
  • UltraVNCViewerPortable
  • UniversalExtractor
  • UnxUtils
  • UPX
  • urlprotocolview
  • usb_format
  • UsbDriveAntiVirus
  • UsbFlashDriveDataRecovery
  • USBLocker
  • USBTrace
  • userassistview
  • userprofilesview
  • uTorrent
  • videocacheview
  • VirtualCdControlPanel
  • VMMap
  • vncpassview
  • VolumeID
  • volumouse
  • VueScan
  • WatermarkingLite
  • webvideocap
  • whatinstartup
  • WhoIs
  • whoistd
  • Windows7Loader
  • winfontsview
  • winlister
  • WinObj
  • WinRAR
  • WinSCP
  • WinSetupFromUSB
  • WinToFlash
  • wirelessnetview
  • wmimgmt
  • wul
  • XLSViewer
  • XMing
  • Yadis
  • ZoomIt

Monday, October 26, 2009

Regular Expressions

I mostly use RegexBuddy to create and test my regular expressions, but I just came across some nice free online tools RegExr , RexV and Nregex to create and test regular expressions. Also useful, a toolbox and some examples of some commonly required regular expressions...

Friday, October 16, 2009

Dropbox - daily usage

Dropbox is a free online service to easily synchronize and backup your data. It can be very useful and it has some very nice features, but it has some limitations too.

Dropbox features:
-2GB free online storage, easy registration.
-Easy sync to different computers / Macs.
-Files are always reachable using the Dropbox website.
-Dropbox keeps 30 days version and deleted files history.
-A public folder with direct static file url's makes it possible to host your site on Dropbox server.
-Sharing of folders between Dropbox accounts is possible.

Dropbox limitation:
-No filtering on files / folders is possible.
-For each account, one folder is synchronized. You need to use Junctions (hardlink NTFS shortcuts (easy Junction explorer extention)) if you want to include external folders in the sync.
-No default support for multiple account synchronization. You need to use the portable Dropbox (download) if you want to synchronize multiple accounts from the same computer at the same time.
-No option to make Dropbox always request for account password on startup (would be useful for the portable version).

Personal usage scenario's:
-Synchronization of my AI Roboform data so I have all my login's everywhere. I combine this with the Roboform2go on my USB stick and Dropbox portable to make sure everything is kept in sync.
-Hosting of files for websites. Instead of having to upload them to different free hosts with ads, I can now use the Dropbox space for easy hosting. Even complete websites are usable when hosted within the 'public' folder of a Dropbox account.
-Combination of FreeOTFE portable to make sure my USB stick portable Dropbox account is kept save. I place all Dropbox files within a FreeOTFE secured file, since else when losing USB stick, anyone could have access to my Dropbox account by just starting up the portable Dropbox application.


Since I use my personal SVN I wanted to make a combination of the automatic Dropbox synchronization coupled to the full control SVN synchronization for development projects. By using the Junctions I could link the SVN folders into Dropbox. Now I have an auto sync of files and folders, but I can manually sync with SVN to have an extra backup and history tracking with full control. The downside of this is that all hidden '.svn' folders are kept in sync too within Dropbox and this can take a lot of your Dropbox space.

Sunday, October 11, 2009

Keep your batteries in good shape

I red a good article on how to keep your batteries in good shape: Dutch, Babelfish translated English version
Top tips:
  • Make sure to recharge long enough before first usage
  • Always use the original or exactly matching charger
  • Nickel Cadmium (NiCd) batteries should be completely empty before recharging (a battery memory effect will shorten the battery live)
    • These batteries are most often used as AAA or AA rechargeable batteries or older mobile phones
  • Lithium Ion batteries should never be completely empty before recharging (no battery memory, but very sensitive to higher temperatures while charging)
    • These batteries are most often used in mobile phones, pda's and notebooks
  • It can help to put your batteries in a cool environment when you won't use them for some time.

Thursday, October 1, 2009

Folder structure creator - Excel VBS

If you need to create a lot of folders and subfolders, Excel_2007.jpgwith some specific structure, it can be usefull if you can use the power of Excel to make up your folder names and structure. All kind of easy and quick formulas can be used, and once the strucuture is set up, you can easily create the empty folder structure with just one click by using this little VBS macro. The base folder used will depend on the location of the Excel file, so make sure it's saved or copied in the correct folder.



Sub CreateFolderStructure()
'Create folder for all vlues in current sheet
'folders will be created in folder where the excel file was saved
'folders will be created from first row, first column, until empty row is found
'Example expected cell structure: (data starting in current sheet, column A, row 1)
'folder1 subfolder1 subsubfolder1
'folder2
'folder3 subfolder3
'...
'this will result in:
'\folder1\subfolder1\subsubfolder1
'\folder2
'\folder3\subfolder3
'...
Set fs = CreateObject("Scripting.FileSystemObject")
For iRow = 1 To 65000
pathToCreate = ActiveWorkbook.Path
For iColumn = 1 To 65000
  currValue = Worksheets(ActiveCell.Worksheet.Name).Cells(iRow, iColumn).Value
  If (currValue = "") Then
   Exit For
  Else
   pathToCreate = pathToCreate & "\" & CStr(currValue)
   'MsgBox (pathToCreate)
   folderToCreate = pathToCreate
   If Not (fs.FolderExists(folderToCreate)) Then
    fs.CreateFolder (folderToCreate)
   End If
  End If
Next
Next
End Sub



ActiveCell.Worksheet.Name: provides the path of the folder where the open Exel sheet resides


CreateObject("Scripting.FileSystemObject") allows to work with folder objects, used to check if a folder exists (fs.FolderExists(<foldername>) ) or create it ( fs.CreateFolder (<foldername>)).


The Excel sheet with the macro can be downloaded here (mirror). Before running the macro make sure the rows and columns of the active sheet are filled in correctly and the Excel file is saved at the correct location. Next simply run the macro by using Men 'Developmen' -> 'Macros' -> CreateFolderStructure -> Run.


foldercreatorexcel1.png foldercreatorexcel2.png


foldercreatorexcel3.png foldercreatorexcel4.png


If you created to many empty folders by accident, you can easily remove them again using this little tool: Remove Empty Directories


Update 13/11/2009: Modified the Excel VBS script to let you navigate to the desired base folder upon launching the macro.

Wednesday, September 16, 2009

Mouse rocker gestures

Mouse Rocker Based on the AutoHotKey script from Adam Pash on the Lifehacker site, I made my own version to fit my personal mouse rocker gesture needs.
Basically, a mouse rocker gesture requires that you press one mouse button, hold it down, then press the other. You can rock across the mouse from right-to-left or left-to-right; each direction you rock gives you a different result. Once you get used to this gesture, the name makes perfect sense, and you'll wonder why you weren't mouse rocking your whole life.
I changed the way the script is configured so one can now easily add process names in the rocker.ini file to change the behaviour for a specific application. And I added extra navigation combinations so one should be able to easily make it fit his own needs. For example in excel you can now easily switch to the next or previous tabbed sheet using mouse rocker gestures. UltraEdit switching between open files is also supported by sending 'Alt + up' or 'Alt + down' keys with the mouse rocker gestures. On first run, it will now also ask if you want to start the tool automatically during windows start. The ini file is created on first run in the directory from where the rocker.exe file is launched. The different groups denote the keys that will be send when a mouse rocker gesture is detected, one can easily add or remove any process name to change the behaviour in a specific application.
[Preferences] CtrlGroup=itunes.exe CtrlTabGroup=notepad++.exe,dreamweaver.exe,pidgin.exe CtrlUpDnGroup=empty CtrlPgUpDnGroup=excel.exe AltGroup=firefox.exe,iexplore.exe,opera.exe,feeddemon.exe,explorer.exe AltUpDnGroup=uedit32.exe AltPgUpDnGroup=empty BckspGroup=empty IgnoreGroup=empty Startup=1 UpdateCheck=0 Version=0.3
No installation is required, just download the exe file (mirror) and run it. The AutoHotKey source script can be downloaded here (mirror).

Hibernate subquery join using Criteria

hibernate I recently needed to create max query in hibernate returning an object instead of the maximal value of the field and I wanted to do this using Hibernate Criteria in our JPA environment. A simple example of what I wanted:
select * from user where userid = (select max(userid)                 from user                 where company = 'aCompanyName')
The way to program this in JPA/Hibernate using Criterias, DetachedCriteria and Subqueries. Make sure to use Subqueries.propertyEq instead of Subqueries.eq if you want to join on a field:
public User getMaxUserOfCompany(String companyName) {  Session session = (Session) em.getDelegate();  DetachedCriteria subCriteria = DetachedCriteria.forClass(User.class);  subCriteria.add(Restrictions.eq("company", companyName));  subCriteria.setProjection(Projections.max("userid") );  Criteria criteria = session.createCriteria(User.class);  criteria.add(Subqueries.propertyEq("userid", subCriteria));  return (User) criteria.uniqueResult(); }

Thursday, July 2, 2009

SpeedTouch ADSL router patch

Recently we had to leave Tele2 with theire very nice unlimited offering. We joined Telenet because we coulnd't connect but through a coax cable in our new house.
For Tele2, we had to buy a ADSL modem + router, the Thomson Alcatel SpeedTouch 716v5. It's a very nice router with Wifi and VOIP integrated. Since this router was still working fine, I didn't want to throw it away. But the problem is that Tele2 puts an administrator password on it, and they didn't want to remove the password or provide it to me. To flash a new rom on the router, you also need to have the administrator password.

I was able to find a method to flash a new rom on the router without having the administrator rights. I flashed this (mirror) rom onto it:
-connect your computer with a lan cable to the SpeedTouch router
-assign a static ip to your network card, you can just copy the settings you got when you received an IP from the DHCP
-start the flashing by executing the upgrade file
-when the upgrade requests for the router password, use a small screwdriver to push the reset button and keep this reset button pressed. In the meantime power off the SpeedTouch and power it on again. Keep on pressing the reset button and wait for 10 - 15 seconds. The power led will blink red, now you can release the reset button.
-Quickly press back on the upgrade flasher, the software will search again for the SpeedTouch device and continue the upgrade without requesting the administrator password.

I had to do it over some times before all went right, but at the end I managed to flash the default SpeedTouch716v5 rom v 6.1.7.2.

Once the default rom is installed, you will have Administrator rights by logging in using: username Administrator and emtpy password.

Next, I configured the router to not use ADSL modem anymore, but use the 4th lan port as incoming internet connection and distribute the internet over Wifi and the remaining 3 lan ports with the build in DHCP server.
To do so, I had to connect to the SpeedTouch through telnet:
-Start -> Run -> type CMD and press execute
-in the command window type: telnet 192.168.1.254 and press enter
-type in your administrator password, default: username Administrator, empty password.
-copy / paste the following commands to execute them on the SpeedTouch device (it requires rom v6.1 to work properly)

:ppp relay flush
:ppp flush
:eth flush
:atm flush
:atm phonebook flush
:eth bridge ifdelete intf=ethport4
:eth ifadd intf=eth_wan
:eth ifconfig intf=eth_wan dest=ethif4
:eth ifattach intf=eth_wan
:ip ifadd intf=ip_wan_eth dest=eth_wan
:ip ifconfig intf=ip_wan_eth status=up
:ip ifattach intf=ip_wan_eth
:nat ifconfig intf=ip_wan_eth translation=enabled
:dhcp client ifadd intf=ip_wan_eth
:dhcp client ifconfig intf=ip_wan_eth metric=5 dnsmetric=5
:dhcp client rqoptions add intf=ip_wan_eth option=dhcp-lease-time
:dhcp client rqoptions add intf=ip_wan_eth option=dhcp-renewal-time
:dhcp client rqoptions add intf=ip_wan_eth option=dhcp-rebinding-time
:dhcp client rqoptions add intf=ip_wan_eth option=subnet-mask
:dhcp client rqoptions add intf=ip_wan_eth option=classless-static-routes
:dhcp client rqoptions add intf=ip_wan_eth option=default-routers
:dhcp client rqoptions add intf=ip_wan_eth option=classfull-static-routes
:dhcp client rqoptions add intf=ip_wan_eth option=domain-name-servers
:dhcp client ifattach intf=ip_wan_eth
:saveall


You can now connect your incoming internet connection on LAN port 4 and the internet will be distributed over the other LAN ports and Wifi. I tested this method with rom version 6.2. But then the internet was only distributed over the 3 remaining LAN ports, and not over the WIFI connections.

Finaly, I configured VOIP. I find out Weepee was very cheap and working very well so far. They have a very fast email support. To get a Weepee fixed line telephone number it costs 9 euros. One time fee! You can chose to receive a new number or transfer your existing fixed number, either way it will cost only 9euros. After this one time cost, you only have to pay for the calls you make, and the rates are very competitive.
To configure Weepee VOIP on my SpeedTouch, I used these VOIP settings:

SIP URI: SIP username provided by Weepee (12 numbers, starting with 32)
Username: SIP username provided by Weepee (12 numbers, starting with 32)
Password: SIP password provided by Weepee
Displayname: SIP username provided by Weepee
Abbreviated number: SIP username provided by Weepee
Port: Phone 1

Expert VOIP configuration:

Registrar: SIP server provided by Weepee, ssw7.weepee.org
Registrar Port: 5060
Proxy: SIP server provided by Weepee, ssw7.weepee.org
Proxy prot: 5060
Expire time: 3600


That should do the trick. I hope someone else can benefit from it.

Update: 8/11/2009

WeePee now has native support for Skype over VOIP. You can be called and make calls for free over Skype. To make it work you'll need to follow these steps:

- Login into WeePee client configuration panel: https://ssl.weepee.org/klanten/flash

- Go to 'Information' -> 'Skype'. As mentioned on this page, you need a Skype Business Account. You'll need to create a new account through the Skype Business Account User management panel using the Skype button 'Create a business account'. (It doesn't work by registering an existing account into Skype Business)

- When the Skype business account is created, you'll need to login once using Skype normally. This way, you'll be able to add your contacts into this account too.

- Now you can provided the Skype username and password of the Skype Business Account into WeePee client configuration panel. Click on the 'Create' button to connect Skype into Weepee. Your status should change to 'Online'.

- To make calls over Skype through WeePee, click on the tab menu 'Short numbers' within the WeePee client configuration panel. For each of your Skype contacts, add a shortcode by adding the 'skype#<skypeusername>' for each shortcode. No you can call your Skype contacts by calling for example '25' from your phone.

Sunday, May 24, 2009

Cobol data + Cobol copybook + Java conversion

Although I'm absolutely not a fan of Cobol, it's still inevitable in the Finance IT development sector.

Lately, we needed a way to let Cobol and Java data work together, and we didn't want to hard code the complete data structure. To make this possible we noticed different non-free applications exist, but the open source project Cb2Xml got our attention.

This project already worked out some Java code to parse a Cobol copybook and convert it into an XML representation. (The copybook can be seen as the interface of the Cobol data). But this project was mend to import and export data from xml to Cobol stream and vice-versa, while we needed some Java objects to work with the received Cobol data input.

So I added some extra code to convert a Cobol data stream (String) into a Java object (using a Hashtable internally). Now, it's possible to provide a String of Cobol data and its interface definition (Cobol copybook file) and return a Java CobolElements object. One can search in this CobolElements object based on the name or xpath. I made a simple example to test, which might make it clear how to use the code.

A simple example of Cobol copybook used as data interface:

 01 GOUTCTT-CSISEQ                                               .
  02 GANSCTT-CSISEQ                                              .
   03 CFND                                 PIC  9(4)             .
   03 GCTB                                                       .
    04 NUPD                                PIC  9(8)             .
    04 GNUMCTB                                                   .
     05 NCTB-12                            PIC  9(12)            .
     05 NCTB-04                            PIC  9(4)             .
     05 CFMT                               PIC  9(2)             .
    04 GCDRCTB                                                   .
     05 CRGOCDRCTB                         PIC  9(1)             .
     05 NROFCMCCDRCTB                      PIC  9(3)             .
     05 GBRACMCCDRCTB                                            .
      06 NBRACMCCDRCTB                     PIC  9(6)             .
      06 GBRACDRCTB                                               
             REDEFINES NBRACMCCDRCTB                             .
       07 NROFCDRCTB                       PIC  9(3)             .
       07 NBRACDRCTB                       PIC  9(3)             .
    04 CBLK                                PIC  9(2)              
             OCCURS 5                                            .
    04 CBLK2                               PIC  9(3)              
             OCCURS 5                                            .

The data that should match this copybook:

input = "00000000000039300022955600000123703602231122334455111222333444555"; 

Converting the data to a Java CobolElements object:

  • Converting the copybook to an xml representation (this XML Document should be created once for each copybook and can be cached):
Document cb2doc = Cb2Xml.convert(new File(_cobolCopybookFileName), _debug); 
  • Converting the Cobol data stream (input string) to it's matching Java representation:
CobolElements cobolElements = Dat2Java.convertWithDoc(input, cb2doc); 

Retrieving data from the CobolElements object:

CobolElement childElement = cobolElements.retrieveChildElement("GOUTCTT-CSISEQ/GANSCTT-CSISEQ/GCTB/GNUMCTB/NCTB-12"); 
System.out.println(childElement.getData()); 
//returns: 393000229556 
childElement = cobolElements.retrieveChildElement("GOUTCTT-CSISEQ/GANSCTT-CSISEQ/GCTB/CBLK[3]"); 
System.out.println(childElement.getData()); 
//returns: 44 

Download source and jar (zip 1,78MB) (link updated 16/09/2009)

Eclipse project

eclipse Each project (workspace) in Eclipse has it's own .project file. To easily open the correct workspace with Eclipse, I wrote a little batch script that can be associated in Windows with the .project files. Now, I only have to double click the .project file and Eclipse will be loaded in the correct workspace.

  • Save the batch script bellow and make sure the correct Eclipse.exe file is used. You might want to change the Eclipse options if required.
  • Double click on a .project file, the first time Windows will ask how to open this file. Choose the .bat batch file script to open your .project file and make sure to make Windows remembers this option.
  • Eclipse will now open with the correct workspace loaded.
@echo off
set ECLIPSE_BIN=R:\tools\eclipse\eclipse.exe
set ECLIPSE_OPTIONS=-refresh -showlocation -Xmx512M -XX:MaxPermSize=512m
set PROJECT_FULL_PATH=%1%
set PROJECT_FOLDER=%PROJECT_FULL_PATH:.project=%
cd %PROJECT_FOLDER%
cd ..
set PROJECT_WORKSPACE="%CD%"
start "eclipse" "%ECLIPSE_BIN%" %ECLIPSE_OPTIONS% -data %PROJECT_WORKSPACE%
@echo on

Download batch script.

Monday, May 11, 2009

LogExpert - Windows tail freeware

Today I tested yet another Windows tail application, LogExpert, and I was very amazed with its features!
It has most of the features I'm used to work with in BareTail Pro, like adding some filters (regex) or highlighting lines matching some text. And unlike WinLogTail, it does support opening multiple files at one with a tabbed interface as in BareTail.
Besides all this, I really appreciate the following functions:
  • run some specified application and providing some information of the selected line. This way, one can easily open the tailed file within UltraEdit on the selected line.
  • parse the timestamps of a log file and show these in separated columns
  • freeze some columns to make these alway be shown and only scrolling other columns (horizontal scrolling)
  • advanced filtering with possibility to show some lines before or after the line matching your filter
  • activate an 'edit' mode to easily copy some parts of a line
  • add some bookmarks and comments on a line to easily find these back afterwards
  • support for multifiles (show files as app.log, app.log1, app.log2, etc as one big file)
  • support for different files provided on the command line while opening, so the scripts we used to open many files with BareTail also work with LogExpert (but opening many files from shared drives is not as lighting fast as with BareTail)

So, compared to BareTail it has many benefits and it's completely free! But one should know, it's not always as fast and stable as BareTail, but so far it worked very well for me.

Some nice screenshots (but those only show a small subset of all it's features):

Sunday, April 26, 2009

Find-Replace text in files from command line

In Linux one can easily find and replace some text in files by using the sed application. In Windows, I couldn’t find a decent build in equivalent. I ended by using the ported version of sed for Windows. I works quite well using the same syntax and doesn’t require to be installed. Just make sure sed.exe, libiconv2.dll and libintl3.dll are existing in the same folder or on the classpath.

One should know any file altered by sed for Windows will result in this file being also converted from Windows line ending (using 2 characters: Carriage Return + Line Feed) to Linux line endings (only 1 character: Line Feed). This can sometime give some strange results. To overcome this, one should make sure the last replacement in the file is set to replace the Line Feeds back to Carriage Return + Line Feed:

sed -i "s/$/\r/" "%DIR%\file.txt"

Some other useful sed commands are available from sed sourcefourge:

FILE SPACING:
 # double space a file
 sed G
 # double space a file which already has blank lines in it. Output file
 # should contain no more than one blank line between lines of text.
 sed '/^$/d;G'
 # triple space a file
 sed 'G;G'
 # undo double-spacing (assumes even-numbered lines are always blank)
 sed 'n;d'
 # insert a blank line above every line which matches "regex"
 sed '/regex/{x;p;x;}'
 # insert a blank line below every line which matches "regex"
 sed '/regex/G'
 # insert a blank line above and below every line which matches "regex"
 sed '/regex/{x;p;x;G;}'
NUMBERING:
 # number each line of a file (simple left alignment). Using a tab (see
 # note on '\t' at end of file) instead of space will preserve margins.
 sed = filename | sed 'N;s/\n/\t/'
 # number each line of a file (number on left, right-aligned)
 sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'
 # number each line of file, but only print numbers if line is not blank
 sed '/./=' filename | sed '/./N; s/\n/ /'
 # count lines (emulates "wc -l")
 sed -n '$='
TEXT CONVERSION AND SUBSTITUTION:
 # IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # works on ssed, gsed 3.02.80 or higher
 # IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$/`echo -e \\\r`/"            # command line under ksh
 sed 's/$'"/`echo \\\r`/"             # command line under bash
 sed "s/$/`echo \\\r`/"               # command line under zsh
 sed 's/$/\r/'                        # gsed 3.02.80 or higher
 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$//"                          # method 1
 sed -n p                             # method 2
 # IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 # Can only be done with UnxUtils sed, version 4.0.7 or higher. The
 # UnxUtils version can be identified by the custom "--text" switch
 # which appears when you use the "--help" switch. Otherwise, changing
 # DOS newlines to Unix newlines cannot be done with sed in a DOS
 # environment. Use "tr" instead.
 sed "s/\r//" infile >outfile         # UnxUtils sed v4.0.7 or higher
 tr -d \r <infile >outfile            # GNU tr version 1.22 or higher
 # delete leading whitespace (spaces, tabs) from front of each line
 # aligns all text flush left
 sed 's/^[ \t]*//'                    # see note on '\t' at end of file
 # delete trailing whitespace (spaces, tabs) from end of each line
 sed 's/[ \t]*$//'                    # see note on '\t' at end of file
 # delete BOTH leading and trailing whitespace from each line
 sed 's/^[ \t]*//;s/[ \t]*$//'
 # insert 5 blank spaces at beginning of each line (make page offset)
 sed 's/^/     /'
 # align all text flush right on a 79-column width
 sed -e :a -e 's/^.\{1,78\}$/ &/;ta'  # set at 78 plus 1 space
 # center all text in the middle of 79-column width. In method 1,
 # spaces at the beginning of the line are significant, and trailing
 # spaces are appended at the end of the line. In method 2, spaces at
 # the beginning of the line are discarded in centering the line, and
 # no trailing spaces appear at the end of lines.
 sed  -e :a -e 's/^.\{1,77\}$/ & /;ta'                     # method 1
 sed  -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/'  # method 2
 # substitute (find and replace) "foo" with "bar" on each line
 sed 's/foo/bar/'             # replaces only 1st instance in a line
 sed 's/foo/bar/4'            # replaces only 4th instance in a line
 sed 's/foo/bar/g'            # replaces ALL instances in a line
 sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
 sed 's/\(.*\)foo/\1bar/'            # replace only the last case
 # substitute "foo" with "bar" ONLY for lines which contain "baz"
 sed '/baz/s/foo/bar/g'
 # substitute "foo" with "bar" EXCEPT for lines which contain "baz"
 sed '/baz/!s/foo/bar/g'
 # change "scarlet" or "ruby" or "puce" to "red"
 sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'   # most seds
 gsed 's/scarlet\|ruby\|puce/red/g'                # GNU sed only
 # reverse order of lines (emulates "tac")
 # bug/feature in HHsed v1.5 causes blank lines to be deleted
 sed '1!G;h;$!d'               # method 1
 sed -n '1!G;h;$p'             # method 2
 # reverse each character on the line (emulates "rev")
 sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
 # join pairs of lines side-by-side (like "paste")
 sed '$!N;s/\n/ /'
 # if a line ends with a backslash, append the next line to it
 sed -e :a -e '/\\$/N; s/\\\n//; ta'
 # if a line begins with an equal sign, append it to the previous line
 # and replace the "=" with a single space
 sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
 # add commas to numeric strings, changing "1234567" to "1,234,567"
 gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta'                     # GNU sed
 sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta'  # other seds
 # add commas to numbers with decimal points and minus signs (GNU sed)
 gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'
 # add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
 gsed '0~5G'                  # GNU sed only
 sed 'n;n;n;n;G;'             # other seds
SELECTIVE PRINTING OF CERTAIN LINES:
 # print first 10 lines of file (emulates behavior of "head")
 sed 10q
 # print first line of file (emulates "head -1")
 sed q
 # print the last 10 lines of a file (emulates "tail")
 sed -e :a -e '$q;N;11,$D;ba'
 # print the last 2 lines of a file (emulates "tail -2")
 sed '$!N;$!D'
 # print the last line of a file (emulates "tail -1")
 sed '$!d'                    # method 1
 sed -n '$p'                  # method 2
 # print the next-to-the-last line of a file
 sed -e '$!{h;d;}' -e x              # for 1-line files, print blank line
 sed -e '1{$q;}' -e '$!{h;d;}' -e x  # for 1-line files, print the line
 sed -e '1{$d;}' -e '$!{h;d;}' -e x  # for 1-line files, print nothing
 # print only lines which match regular expression (emulates "grep")
 sed -n '/regexp/p'           # method 1
 sed '/regexp/!d'             # method 2
 # print only lines which do NOT match regexp (emulates "grep -v")
 sed -n '/regexp/!p'          # method 1, corresponds to above
 sed '/regexp/d'              # method 2, simpler syntax
 # print the line immediately before a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{g;1!p;};h'
 # print the line immediately after a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{n;p;}'
 # print 1 line of context before and after regexp, with line number
 # indicating where the regexp occurred (similar to "grep -A1 -B1")
 sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
 # grep for AAA and BBB and CCC (in any order)
 sed '/AAA/!d; /BBB/!d; /CCC/!d'
 # grep for AAA and BBB and CCC (in that order)
 sed '/AAA.*BBB.*CCC/!d'
 # grep for AAA or BBB or CCC (emulates "egrep")
 sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d    # most seds
 gsed '/AAA\|BBB\|CCC/!d'                        # GNU sed only
 # print paragraph if it contains AAA (blank lines separate paragraphs)
 # HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
 # print paragraph if it contains AAA and BBB and CCC (in any order)
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
 # print paragraph if it contains AAA or BBB or CCC
 sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
 gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d'         # GNU sed only
 # print only lines of 65 characters or longer
 sed -n '/^.\{65\}/p'
 # print only lines of less than 65 characters
 sed -n '/^.\{65\}/!p'        # method 1, corresponds to above
 sed '/^.\{65\}/d'            # method 2, simpler syntax
 # print section of file from regular expression to end of file
 sed -n '/regexp/,$p'
 # print section of file based on line numbers (lines 8-12, inclusive)
 sed -n '8,12p'               # method 1
 sed '8,12!d'                 # method 2
 # print line number 52
 sed -n '52p'                 # method 1
 sed '52!d'                   # method 2
 sed '52q;d'                  # method 3, efficient on large files
 # beginning at line 3, print every 7th line
 gsed -n '3~7p'               # GNU sed only
 sed -n '3,${p;n;n;n;n;n;n;}' # other seds
 # print section of file between two regular expressions (inclusive)
 sed -n '/Iowa/,/Montana/p'             # case sensitive
SELECTIVE DELETION OF CERTAIN LINES:
 # print all of file EXCEPT section between 2 regular expressions
 sed '/Iowa/,/Montana/d'
 # delete duplicate, consecutive lines from a file (emulates "uniq").
 # First line in a set of duplicate lines is kept, rest are deleted.
 sed '$!N; /^\(.*\)\n\1$/!P; D'
 # delete duplicate, nonconsecutive lines from a file. Beware not to
 # overflow the buffer size of the hold space, or else use GNU sed.
 sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
 # delete all lines except duplicate lines (emulates "uniq -d").
 sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
 # delete the first 10 lines of a file
 sed '1,10d'
 # delete the last line of a file
 sed '$d'
 # delete the last 2 lines of a file
 sed 'N;$!P;$!D;$d'
 # delete the last 10 lines of a file
 sed -e :a -e '$d;N;2,10ba' -e 'P;D'   # method 1
 sed -n -e :a -e '1,10!{P;N;D;};N;ba'  # method 2
 # delete every 8th line
 gsed '0~8d'                           # GNU sed only
 sed 'n;n;n;n;n;n;n;d;'                # other seds
 # delete lines matching pattern
 sed '/pattern/d'
 # delete ALL blank lines from a file (same as "grep '.' ")
 sed '/^$/d'                           # method 1
 sed '/./!d'                           # method 2
 # delete all CONSECUTIVE blank lines from file except the first; also
 # deletes all blank lines from top and end of file (emulates "cat -s")
 sed '/./,/^$/!d'          # method 1, allows 0 blanks at top, 1 at EOF
 sed '/^$/N;/\n$/D'        # method 2, allows 1 blank at top, 0 at EOF
 # delete all CONSECUTIVE blank lines from file except the first 2:
 sed '/^$/N;/\n$/N;//D'
 # delete all leading blank lines at top of file
 sed '/./,$!d'
 # delete all trailing blank lines at end of file
 sed -e :a -e '/^\n*$/{$d;N;ba' -e '}'  # works on all seds
 sed -e :a -e '/^\n*$/N;/\n$/ba'        # ditto, except for gsed 3.02.*
 # delete the last line of each paragraph
 sed -n '/^$/{p;h;};/./{x;/./p;}'
SPECIAL APPLICATIONS:
 # remove nroff overstrikes (char, backspace) from man pages. The 'echo'
 # command may need an -e switch if you use Unix System V or bash shell.
 sed "s/.`echo \\\b`//g"    # double quotes required for Unix environment
 sed 's/.^H//g'             # in bash/tcsh, press Ctrl-V and then Ctrl-H
 sed 's/.\x08//g'           # hex expression for sed 1.5, GNU sed, ssed
 # get Usenet/e-mail message header
 sed '/^$/q'                # deletes everything after first blank line
 # get Usenet/e-mail message body
 sed '1,/^$/d'              # deletes everything up to first blank line
 # get Subject header, but remove initial "Subject: " portion
 sed '/^Subject: */!d; s///;q'
 # get return address header
 sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
 # parse out the address proper. Pulls out the e-mail address by itself
 # from the 1-line return address header (see preceding script)
 sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
 # add a leading angle bracket and space to each line (quote a message)
 sed 's/^/> /'
 # delete leading angle bracket & space from each line (unquote a message)
 sed 's/^> //'
 # remove most HTML tags (accommodates multiple-line tags)
 sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
 # extract multi-part uuencoded binaries, removing extraneous header
 # info, so that only the uuencoded portion remains. Files passed to
 # sed must be passed in the proper order. Version 1 can be entered
 # from the command line; version 2 can be made into an executable
 # Unix shell script. (Modified from a script by Rahul Dhesi.)
 sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode   # vers. 1
 sed '/^end/,/^begin/d' "$@" | uudecode                    # vers. 2
 # sort paragraphs of file alphabetically. Paragraphs are separated by blank
 # lines. GNU sed uses \v for vertical tab, or any unique char will do.
 sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
 gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'
 # zip up each .TXT file individually, deleting the source file and
 # setting the name of each .ZIP file to the basename of the .TXT file
 # (under DOS: the "dir /b" switch returns bare filenames in all caps).
 echo @echo off >zipup.bat
 dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat

End (kill) some application from command line

Different methods exist in Windows to end or kill some application from command line.

The normal way to do this is by using the TASKKILL command, for example to end FireFox one can use:

TASKKILL /F /IM “firefox.exe”

When you need to stop some command window it can be dangerous to just kill any cmd.exe, so one could use the window title to kill a specific batch window, for example to kill any batch window with name starting with “SIM-” I used:

TASKKILL /FI "WINDOWTITLE eq SIM-*"

But sometimes you’ll need to be more specific, I wanted an automated way to kill some Java application but only when it was started with some specific parameter. The only way I found to do so on Windows was by using the WMIC commands. This example show how I killed our Java monitor application which started as follows:

java -Dapp.prop.file.monitor="monitor_system.properties" -classpath SOME_JARS Monitor

Kill this specific Java instance:

wmic process where "name like '%%java.exe%%' AND commandline like '%%monitor%%'" DELETE

Before killing your application, you want to make sure your query will result in the correct process, one can easily check the query by executing the similar wmic command without yet killing the process:

wmic process where "name like '%%java.exe%%' AND commandline like '%%monitor%%'" get processid, name, commandline /VALUE

More examples and combinations that can be used with wmic are available on this page.

The Linux equivalent we used was pkill, for example to kill our simulators we used:

pkill -f "simulator.jar"

Sunday, April 19, 2009

SVN Automated repeated backup / dump – Windows / Linux

Every project using SVN should have an automated backup system in place for their SVN repository. We decided we wanted a weekly incremental backup. But we also wanted to easily create a new initial backup. I scripted it on Windows and Linux with the Shell/Batch scripts shown below.

For both scripts, the same methods is used: we keep track of the last backed up revision in a file called svn_backup_delta.txt. Every time the script is started, an incremental dump is exported and archived, starting from the last dumped revision till the latest available revision. If the latest revision is the same as the last dumped revision, nothing will happen.

If the svn_backup_delta.txt doesn’t exist or the revision could not be read correctly, a complete initial dump from revision 0 till the latest revision will be exported and archived. The scripts are called daily or weekly with a standard OS scheduler. When we want to have a new initial complete dump, we simply remove the svn_backup_delta.txt file.

The scripts can be a good starting point to create your own automated SVN dumping mechanism.

Windows svnbackup.bat Batch script:

@echo off
::CONFIGURATION
  SET REPO_URL=https://url:8443/base/project
  SET REPO_PATH=C:\repository\project
  SET DUMP_DIR=C:\backup
  SET WINRAR=C:\Program Files\WinRAR\Rar.exe
  SET LAST_DELTA_FILE=%CD%\svn_backup_delta.txt
:END_CONFIGURATION
echo.
echo Starting SVN backup
echo Using following configuration:
echo  -Repository URL: "%REPO_URL%"
echo  -Repository full path: "%REPO_PATH%"
echo  -File to keep track of last revision backed up: %LAST_DELTA_FILE%
SET DUMP_FILE=""
SET Today=%Date: =0%
SET Today=%Today:~-4%%Date:~-7,2%%Date:~-10,2%
SET Now=%Time: =0%
FOR /F "tokens=1,2 delims=:.," %%A IN ("%Now%") DO SET Now=%%A%%B
mkdir "%DUMP_DIR%"
svn info "%REPO_URL%" > %TEMP%.\repoinfo.tmp
SET LAST_REV=
for /f "usebackq tokens=1,2 delims=: " %%i in (`find "Revision: " "%TEMP%\repoinfo.tmp"`) do (
  SET LAST_REV=%%j
)
echo Last revision:%LAST_REV%
echo Last delta file: %LAST_DELTA_FILE%
IF EXIST "%LAST_DELTA_FILE%" GOTO DELTA
:FULL
  SET DUMP_FILE_NAME=%Today%-%Now%-INITIAL-R%LAST_REV%.dump
  SET DUMP_FILE=%DUMP_DIR%\%DUMP_FILE_NAME%
  echo Dump file name: %DUMP_FILE_NAME%
  echo Dump file: %DUMP_FILE% 
  svnadmin dump "%REPO_PATH%" -r 0:%LAST_REV% > "%DUMP_FILE%"
  "%WINRAR%" m "%DUMP_FILE%.rar" "%DUMP_FILE%"
  echo Date: %Today%-%Now% > "%LAST_DELTA_FILE%"
  echo Revision: %LAST_REV% > "%LAST_DELTA_FILE%"
  goto END
:DELTA
  SET LAST_DELTA=
  for /f "usebackq tokens=1,2 delims=: " %%i in (`find "Revision: " "%LAST_DELTA_FILE%"`) do (
    SET LAST_DELTA=%%j
  )
  echo Last delta:%LAST_DELTA%
  echo Last revision:%LAST_REV%
  
  IF "%LAST_DELTA%"=="" GOTO FULL
  IF "%LAST_DELTA%"=="%LAST_REV%" GOTO END
  SET /A INCRE_LAST_DELTA=LAST_DELTA+1
  SET DUMP_FILE_NAME=%Today%-%Now%-INCREMENTAL-R%INCRE_LAST_DELTA%-TO-R%LAST_REV%.dump
  SET DUMP_FILE=%DUMP_DIR%\%DUMP_FILE_NAME%
  echo Dump file name: %DUMP_FILE_NAME%
  echo Dump file: %DUMP_FILE%
  svnadmin dump "%REPO_PATH%" -r %INCRE_LAST_DELTA%:%LAST_REV% --incremental --deltas > "%DUMP_FILE%"
  "%WINRAR%" m "%DUMP_FILE%.rar" "%DUMP_FILE%"
  echo Date: %Today%-%Now% > "%LAST_DELTA_FILE%"
  echo Revision: %LAST_REV% > "%LAST_DELTA_FILE%"
  goto END
:END
  rem pause
  rem exit
:END

Linux svnbackup.sh Shell script:

#!/bin/sh
#SETUP
REPO_URL=https://url:8443/base/project
REPO_PATH=c:/repository/project
DUMP_DIR=/home/user/dumps
LAST_DELTA_FILE=svn_backup_delta.txt
#BEGINING OF THE SCRIPT ITSELF
date=$(date +%Y%m%d)
LOG=dump-$date.log
LAST_REV=$(svn info $REPO_URL |grep Revision |gawk ' { print $2}')
#Both files MUST exist, otherwise, wer'e going to init new "delting"
if [ -e $LAST_DELTA_FILE ] ; then
  LAST_DELTA=$(cat $LAST_DELTA_FILE)
  if [ $LAST_DELTA -ne $LAST_REV ] ; then
    DUMP_FILE_NAME=$date-INCREMENTAL-R$((LAST_DELTA+1))-TO-R$LAST_REV.dump.gz
    DUMP_FILE=$DUMP_DIR/$DUMP_FILE_NAME
    svnadmin dump $REPO_PATH -r $((LAST_DELTA +1)):$LAST_REV --incremental --deltas 2>>$LOG |gzip -c > $DUMP_FILE
  else
    echo nothing to do : last revision number is same as last delta number, run it again later !
    exit 0
  fi
else
  DUMP_FILE_NAME=$date-INITIAL-R$LAST_REV.dump.gz
  DUMP_FILE=$DUMP_DIR/$DUMP_FILE_NAME
  svnadmin dump $REPO_PATH 2>>$LOG | gzip -c > $DUMP_FILE
fi
#Is all ok ?
if [ $? -eq 0 ] ; then
  rm $DUMP_FILE
  echo -n $LAST_REV > $LAST_DELTA_FILE
else
  echo "ERROR :" $date "Problem occured while saving dump" >>$LOG
fi

Sunday, April 12, 2009

Run custom application or script as a Windows service

If you need your application or script to be started, whenever Windows is booted, you need to install it as a service (so  it will even be started when you didn’t login yet). Microsoft offers a little tool to easily create your own service and make it run some batch file / application as a part of their Windows Server Resource Kit Tools. From this package, you only need srvany.exe and instsrv.exe to install your custom service, but you still need to perform all this manual configuration before your service will be running.

I created the following batch script to easily install or uninstall your custom application as a service. All required information will be requested interactively to the user step by step. No further manual configuration will be needed anymore. Errors will be shown with colors and attention points will be flashed.

@echo off
echo.
echo.
echo Use this script to install / uninstall your custom applications or scripts as a service and make sure they will be started whenever Windows is booted.
SET DFAULT_BACKGROUND_COLOR=0
SET DEFAULT_TEXT_COLOR=7
SET ERROR_BACKGROUND_COLOR=4
SET ERROR_TEXT_COLOR=F
SET ATTENTION_BACKGROUND_COLOR=0
SET ATTENTION_TEXT_COLOR=F
rem 0 = Black
rem 1 = Blue
rem 2 = Green
rem 3 = Aqua
rem 4 = Red
rem 5 = Purple
rem 6 = Yellow
rem 7 = White
rem 8 = Gray
rem 9 = Light Blue
rem A = Light Green
rem B = Light Aqua
rem C = Light Red
rem D = Light Purple
rem E = Light Yellow
rem F = Bright White
:MAIN_MENU
  COLOR %DFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  
  SET RETURN_DRAW_ATTENTION_POINT=MAIN_MENU_INPUT
  GOTO DRAW_ATTENTION
  :MAIN_MENU_INPUT
    echo.
    echo.
    SET INPUT=
    echo To Install a new custom service, press I.
    echo To Uninstall a previously created custom service, press U.
    echo To open the Windows Services application, press S.
    set /p INPUT=Press E to Exit: 
    if /i "%INPUT%" == "i" goto START_INSTALL
    if /i "%INPUT%" == "u" goto START_UNINSTALL
    if /i "%INPUT%" == "s" goto START_SERVICES
    if /i "%INPUT%" == "e" goto EXIT
    GOTO MAIN_MENU
  :END_MAIN_MENU_INPUT
  echo.     
:END_MAIN_MENU
  
:START_INSTALL
  rem INTERACTIVE MODE is used, lines below kept for reference
  :HARD_CODED_PATHS_AND_NAMES
  rem APP_NAME=REPLACE_THIS_WITH_YOUR_CUSTOM_APP_NAME
  rem APP_PATH=FULL_PATH_TO_CUSTOM_APP
  rem one can use %CD% in the path, which will be replace by the current directory
  rem uncomment following 2 lines if you want to use the folder name (where this bat file is run) as service name APP_NAME
  rem SET APP_NAME=
  rem for %%* in (.) do set APP_NAME=%%~n*
  
  :INTERACTIVE_INPUT_OF_PATHS_AND_NAMES
    :INTERACTIVE_INPUT_APP_NAME
      echo.
      SET APP_NAME=
      set /p APP_NAME=Provide the name of your custom application (will be used as the name of the service): 
      echo. 
    :END_INTERACTIVE_INPUT_APP_NAME
    
    :INTERACTIVE_INPUT_OF_APP_PATH
      echo. 
      SET APP_PATH=
      set /p APP_PATH=Provide the full path of your custom application to run as a service (don't use quotes): 
      echo. 
    :END_INTERACTIVE_INPUT_OF_APP_PATH
    
    :CHECK_CUSTOM_APP_PATH
      IF EXIST "%APP_PATH%" GOTO END_CHECK_CUSTOM_APP_PATH
      echo.
      echo Your custom application %APP_NAME% could not be found at %APP_PATH%!
      echo Please try again!
      SET RETURN_DRAW_ERROR_ATTENTION_POINT=INTERACTIVE_INPUT_OF_APP_PATH
      GOTO DRAW_ERROR_ATTENTION
    :END_CHECK_CUSTOM_APP_PATH
      COLOR %DFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
      
    :INTERACTIVE_INPUT_APP_FOLDER
      echo.
      SET APP_FOLDER=
      set /p APP_FOLDER=Provide the folder to be used as working directory for your custom application (press ENTER if not required) (don't use quotes): 
      echo. 
    :END_INTERACTIVE_INPUT_APP_PARAM
      
    :INTERACTIVE_INPUT_APP_PARAM
      echo.
      SET APP_PARAM=
      set /p APP_PARAM=Provide the parameters of your custom application (press ENTER if no parameters are required) (don't use quotes): 
      echo. 
    :END_INTERACTIVE_INPUT_APP_PARAM
    
    :INTERACTIVE_INPUT_OF_INST_SERVICE_APP_PATH
      echo. 
      SET INST_SERVICE_APP_PATH=
      set /p INST_SERVICE_APP_PATH=Provide the full path to instsrv.exe (don't use quotes). LEAVE EMPTY to use default path (%CD%\instsrv.exe): 
    :END_INTERACTIVE_INPUT_OF_INST_SERVICE_APP_PATH
    
    :CHECK_INST_SERVICE_APP_PATH_INPUT
      if "%INST_SERVICE_APP_PATH%"=="" goto DEFAULT_INST_SERVICE_APP_PATH
    :END_CHECK_INST_SERVICE_APP_PATH_INPUT
    
    :CHECK_INST_SERVICE_APP_PATH
      IF EXIST "%INST_SERVICE_APP_PATH%" GOTO END_CHECK_INST_SERVICE_APP_PATH
      echo.
      echo instsrv.exe could not be found at %INST_SERVICE_APP_PATH%!
      echo Please try again!
      SET RETURN_DRAW_ERROR_ATTENTION_POINT=INTERACTIVE_INPUT_OF_INST_SERVICE_APP_PATH
      GOTO DRAW_ERROR_ATTENTION
    :END_CHECK_INST_SERVICE_APP_PATH
      COLOR %DFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
    
    :INTERACTIVE_INPUT_OF_SERVICE_APP_PATH
      echo. 
      SET SERVICE_APP_PATH=
      set /p SERVICE_APP_PATH=Provide the full path to srvany.exe (don't use quotes). LEAVE EMPTY to use default path (%CD%\srvany.exe): 
    :END_INTERACTIVE_INPUT_OF_SERVICE_APP_PATH
    
    :CHECK_SERVICE_APP_PATH_INPUT
      if "%SERVICE_APP_PATH%"=="" goto DEFAULT_SERVICE_APP_PATH
    :END_CHECK_SERVICE_APP_PATH_INPUT
    
    :CHECK_SERVICE_APP_PATH
      IF EXIST "%SERVICE_APP_PATH%" GOTO END_CHECK_SERVICE_APP_PATH
      echo.
      echo srvany.exe could not be found at %SERVICE_APP_PATH%!
      echo Please try again!
      SET RETURN_DRAW_ERROR_ATTENTION_POINT=INTERACTIVE_INPUT_OF_SERVICE_APP_PATH
      GOTO DRAW_ERROR_ATTENTION
    :END_CHECK_SERVICE_APP_PATH
      COLOR %DFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  
  :END_INTERACTIVE_INPUT_OF_PATHS_AND_NAMES  
  
  :SHOW_CONFIG
    echo.
    echo Ready to install your custom application, using the following configuration:
    echo.
    echo   Custom application name: %APP_NAME%
    echo   Custom application full path: %APP_PATH%
    echo   Custom application parameters: %APP_PARAM%
    echo   Custom application working directory: %APP_FOLDER%
    echo   srvany.exe full path: %SERVICE_APP_PATH%
    echo   instsrv.exe full path: %INST_SERVICE_APP_PATH%
    echo.
    SET RETURN_DRAW_ATTENTION_POINT=SHOW_CONFIG_WAIT
    GOTO DRAW_ATTENTION
    :SHOW_CONFIG_WAIT
      COLOR %ATTENTION_BACKGROUND_COLOR%%ATTENTION_TEXT_COLOR%
      set INPUT=
      set /p INPUT=Press ENTER to continue installation, all other will return to main menu: 
      if /i not "%INPUT%" == "" GOTO MAIN_MENU
      COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
    :END_SHOW_CONFIG_WAIT
  :END_SHOW_CONFIG
  
  :INSTALL_SERVICE
    "%INST_SERVICE_APP_PATH%" "%APP_NAME%" "%SERVICE_APP_PATH%"
    echo.
    echo Service installed : %APP_NAME%
    echo.
  :END_INSTALL_SERVICE
  
  :UPDATE_REGISTRY
    REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\%APP_NAME%\Parameters" /v "Application" /t REG_SZ /d "%APP_PATH%" /f
    IF NOT "%APP_FOLDER%"=="" REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\%APP_NAME%\Parameters" /v "AppDirectory" /t REG_SZ /d "%APP_FOLDER%" /f
    IF NOT "%APP_PARAM%"=="" REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\%APP_NAME%\Parameters" /v "AppParameters" /t REG_SZ /d "%APP_PARAM%" /f
    echo.
    echo Registry updated to run %APP_PATH% with service %APP_NAME%
    echo.
  :END_UPDATE_REGISTRY
  
  :START_SERVICE
    SET RETURN_DRAW_ATTENTION_POINT=START_SERVICE_INPUT_REQUEST
    GOTO DRAW_ATTENTION
    :START_SERVICE_INPUT_REQUEST
      echo The service is now installed successfully (if no errors are logged above!)
      set INPUT=
      set /p INPUT=To start the service now, press Y. All others will return to main menu: 
      if /i not "%INPUT%" == "y" goto END_START_SERVICE
    :END_START_SERVICE_INPUT_REQUEST
    NET START "%APP_NAME%"
    echo.
    echo Service %APP_NAME% started
  :END_START_SERVICE
  
  GOTO END_START_INSTALL
  
  :DEFAULT_INST_SERVICE_APP_PATH
    echo Using default path for instsrv.exe %INST_SERVICE_APP_PATH%
    SET INST_SERVICE_APP_PATH=%CD%\instsrv.exe
    GOTO END_CHECK_INST_SERVICE_APP_PATH_INPUT
  :END_DEFAULT_INST_SERVICE_APP_PATH
  
  :DEFAULT_SERVICE_APP_PATH
    echo Using default path for srvany.exe %SERVICE_APP_PATH%
    SET SERVICE_APP_PATH=%CD%\srvany.exe
    GOTO END_CHECK_SERVICE_APP_PATH_INPUT
  :END_DEFAULT_SERVICE_APP_PATH
:END_START_INSTALL
  GOTO MAIN_MENU
:START_SERVICES
  start "Services" services.msc
:END_START_SERVICES
  GOTO MAIN_MENU
:START_UNINSTALL
  :INTERACTIVE_INPUT_APP_NAME_UNINSTALL
    echo.
    SET APP_NAME=
    set /p APP_NAME=Provide the name of your custom application (the name of the service to uninstall): 
    echo. 
  :END_INTERACTIVE_INPUT_APP_NAME_UNINSTALL
    
  :INTERACTIVE_INPUT_OF_INST_SERVICE_APP_PATH_UNINSTALL
    echo. 
    SET INST_SERVICE_APP_PATH=
    set /p INST_SERVICE_APP_PATH=Provide the full path to instsrv.exe (don't use quotes). LEAVE EMPTY to use default path (%CD%\instsrv.exe): 
  :END_INTERACTIVE_INPUT_OF_INST_SERVICE_APP_PATH_UNINSTALL
  
  :CHECK_INST_SERVICE_APP_PATH_INPUT_UNINSTALL
    if "%INST_SERVICE_APP_PATH%"=="" goto DEFAULT_INST_SERVICE_APP_PATH_UNINSTALL
  :END_CHECK_INST_SERVICE_APP_PATH_INPUT_UNINSTALL
  
  :CHECK_INST_SERVICE_APP_PATH_UNINSTALL
    IF EXIST "%INST_SERVICE_APP_PATH%" GOTO END_CHECK_INST_SERVICE_APP_PATH_UNINSTALL
    echo.
    echo instsrv.exe could not be found at %INST_SERVICE_APP_PATH%!
    echo Please try again!
    SET RETURN_DRAW_ERROR_ATTENTION_POINT=INTERACTIVE_INPUT_OF_INST_SERVICE_APP_PATH_UNINSTALL
    GOTO DRAW_ERROR_ATTENTION
  :END_CHECK_INST_SERVICE_APP_PATH_UNINSTALL
    COLOR %DFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  
  :STOP_SERVICE
    NET STOP %APP_NAME%
    echo.
    echo Service %APP_NAME% stopped
    echo.
  :END_STOP_SERVICE
  
  :REMOVE_SERVICE
    "%INST_SERVICE_APP_PATH%" %APP_NAME% REMOVE
    echo.
    echo Service %APP_NAME% removed
    echo.
  :END_REMOVE_SERVICE
  
  :UPDATE_REGISTRY_UNINSTALL
    REG DELETE HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\%APP_NAME%\ /va /f
    echo.
    echo Registry cleaned up for service %APP_NAME%
    echo.
  :END_UPDATE_REGISTRY_UNINSTALL
  
  GOTO END_START_UNINSTALL
  
  :DEFAULT_INST_SERVICE_APP_PATH_UNINSTALL
    echo Using default path for instsrv.exe %INST_SERVICE_APP_PATH%
    SET INST_SERVICE_APP_PATH=%CD%\instsrv.exe
    GOTO END_CHECK_INST_SERVICE_APP_PATH_INPUT_UNINSTALL
  :END_DEFAULT_INST_SERVICE_APP_PATH_UNINSTALL
:END_START_UNINSTALL
  GOTO MAIN_MENU
  
:EXIT_WITH_PAUSE
  echo The end. Comments and updates: http://myTselection.blogspot.com, Copyright MightyMouse. Copyright srvany and instsrv Microsoft.
  pause
  GOTO EXIT
:END_EXIT_WITH_PAUSE
  
:EXIT
  exit
:END_EXIT
:DRAW_ATTENTION
  COLOR %ATTENTION_BACKGROUND_COLOR%%ATTENTION_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %ATTENTION_BACKGROUND_COLOR%%ATTENTION_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %ATTENTION_BACKGROUND_COLOR%%ATTENTION_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
:END_DRAW_ATTENTION
  GOTO %RETURN_DRAW_ATTENTION_POINT%
:DRAW_ERROR_ATTENTION
  COLOR %ERROR_BACKGROUND_COLOR%%ERROR_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %ERROR_BACKGROUND_COLOR%%ERROR_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %ERROR_BACKGROUND_COLOR%%ERROR_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %DEFAULT_BACKGROUND_COLOR%%DEFAULT_TEXT_COLOR%
  @PING 1.1.1.1 -n 1 -w 100 >NUL
  COLOR %ERROR_BACKGROUND_COLOR%%ERROR_TEXT_COLOR%
:END_ERROR_DRAW_ATTENTION
  GOTO %RETURN_DRAW_ERROR_ATTENTION_POINT%
@echo on

A full package to easily install your own custom application as a service can be downloaded here, sources (zip).

Update (26/04/09): Added possibility to specify application working directory and command line parameters. (AppDirectory and AppParameter registry keys.)

My “Firefox add-ons” selection

Firefox has one big benefit: it’s support for Add-ons / Plug- ins / Themes / Extensions. Without this benefit, I simply wouldn’t use Firefox, but only Google Chrome.

I list below, the Add-ons I’m using mostly:

  • AI Roboform toolbar: best way to safely keep track of your website logins, passwords, forms. Even better with it’s nice and easy synchronization to Windows Mobile PDA. the nice toolbar allows you to navigate to pages you use most and immediately login, with just 2 or 3 characters to type!
  • All-in-one Gestures extension: easily navigate using your mouse (navigate using mouse movements and special mouse click combinations)
  • Cooliris picture/movie viewer: great way to watch pictures from many sites (facebook, flickr, google images, youtube, …). The new beta 1.10 version has even better support for facebook.
  • Delicious bookmarks: keep your bookmarks online, so you can easily find back what you liked before, everywhere, anytime. This plug-in has a nice sidebar and easy integration.
  • Facebook toolbar: some extra’s for facebook
  • Fetch Text URL: select some non clickable or miss-spelled url and easily open the url in a new tab.
  • Flashgot: integration of Flashget (download manager) in Firefox.
  • Firebug: required add-on for every web developer. Nice JavaScript debugging possibilities.
  • Greasemonkey: framework to easily install scripts to customize your navigation in specific websites. Thousands of scripts are available for many well known sites. Some of the scripts I prefer:
  • IE Tab: some sites still require Internet Explorer. With this plug-in you can combine Internet Explorer within Firefox.
  • NextPlease: add some keyboard and mouse shortcuts to easily navigate through sites with ‘next’ and ‘previous’ kind of links.
  • SendTo: Send selected text to a text file, together with url and timestamp. SendTo v0.4 is not compatible with Firefox 3.0, but I just changed the compatibility settings in the plug-in and it still works very well now. I use this plugin a lot to always keep a text file with some information on applications I download. I download a lot of software and want to be able to easily retrieve what I’ve downloaded before or get some basic information on the tool. I made an UltraEdit macro to copy the text from the sendto file into a new file, this way I can even retrieve all I need even faster in the format I prefer. All files (SendTo v0.4 for FF 3, batch file to start ultra edit with macro, UltraEdit macro) can be downloaded here.
  • Tab Mix Plus: add extra options and extra functionality for the tabs in Firefox. For example duplicate current tab, open tab on mouse hover, show tab close button on mouse hover, etc.
  • WebDeveloper: required add-on for every web developer. See and change many site settings, form settings, CSS settings, javascripts etc.
  • Vista-aero theme: just a nice look and feel, copied from Internet Explorer.

Once Google Chrome will support all this functionality I’ll dump Firefox and will finally get my 400MB stolen memory back.

UPDATE 05/10/2009: added links to my own Greasemonkey scripts

JMeter to check files and file content

Recently, we needed a JMeter which could perform some tests to make sure files were created on the file system,  matching a specific file path and filename, and some text had to match in the file itself. These files were created by a simulator, but to test our project, we wanted to automatically check if the simulator received all required files.

Apparently, JMeter is not often used for such a checks on the file system, but we preferred JMeter so we could combine the result reports nicely with the reports of the sending of the files (performed by another JMeter).

We succeeded by using BeanShell scripts in the JMeter with a BeanShell Sampler and a BeanShell Assertion. I created a template JMeter jmx and replaced some tags (@…@) in this template for each of our test cases. An example of a filled in JMeter jmx file based on this template is available here. We transform the template to the specific test case jmx file using Excel macros for easy configuration management.

In the JMeter we always linked a CheckFileSampler.bsh and CheckFileAssertion.bsh scripts to perform the required checks. Unlimited properties can be assigned in the JMeter jmx and these properties can very easily be retrieved inside the BeanShell script by using:

value = vars.get("<variable name>");

In the BeanShell script, one can just use standard Java code. I made a simplified version of the code, which is less related to our project. In this code, I build up the path of the file which is expected to exist. The file path is build up based on the properties set in the jmx file. I check if the file exists. If it could be found, I match some patterns against the content of the file. If all goes well, the test case succeeds, else it fails.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.io.*;
import java.util.regex.*;
Failure=false;
FailureMessage="";
String testcase = vars.get("testcase");
String simulatorReceiveMessageBasePath = vars.get("simulatorReceiveMessageBasePath");
String partnerName = vars.get("partnerNameProperty");
//should the file exist (be received) or not for this partner? 
String partnerReceiveMessageRequired = vars.get(partnerName + "_partnerRequiredReceiveProperty");
//the following pattern should match within the file (if it exists), semicolon (;) separeted list
String patternsToMatchList = vars.get("patternsToMatchList");
//within the basepath, the simulator creates a new folder on every start of the simulator, we wanted to retrieve the newest folder
//retrieve newest folder from partner
File basefolder = new File(simulatorReceiveMessageBasePath);
if (!basefolder.isDirectory()) {
  Failure = true;
  log.error(testcase + " - Partner " + partnerName  + " incorrect base folder: " + simulatorReceiveMessageBasePath);
  FailureMessage = testcase + ";" + partnerName  + ";INCORRECT_BASE_FOLDER;" + simulatorReceiveMessageBasePath + "|" + FailureMessage;
  break;
}  
File[] foldersinbasefolder = basefolder.listFiles();
Comparator filecomp = new Comparator() {
      public int compare(Object o1, Object o2) {
        return new Long(((File)o2).lastModified()).compareTo
             (new Long(((File) o1).lastModified()));
      }
    };  
Arrays.sort( foldersinbasefolder, filecomp);    
if (foldersinbasefolder == null || foldersinbasefolder.length == 0) {
  Failure = true;
  log.error(testcase + " - Partner " + partnerName  + " no subfolder could be found in the base folder: " + simulatorReceiveMessageBasePath);
  FailureMessage = testcase + ";" + partnerName  + ";NOSUBFOLDERSINBASEFOLDER;" + simulatorReceiveMessageBasePath + "|" + FailureMessage;
  break;
}    
String partnerSimulatorFolder  = foldersinbasefolder[0].getCanonicalPath();
String partnerReceiveMessageFullPath = partnerSimulatorFolder + "\\" + testcase + "_message.xml";
if (partnerReceiveMessageRequired != null && partnerReceiveMessageRequired.equals("1"))  {
      
  //partner should have received the message
  log.info(testcase + " - Partner " + partnerName + " should have received message with full path " + partnerReceiveMessageFullPath);
  File f = new File(partnerReceiveMessageFullPath);
  if (f.exists() && f.isFile() && f.length() > 0) {
    log.info(testcase + " - Partner " + partnerName + " has received message with full path " + partnerReceiveMessageFullPath + " size: " + f.length() + "bytes");
      
    int patternsMatched = 0;
    if (patternsToMatchList != null && patternsToMatchList != "") {
      
      //check if the pattern can be matched with the file
      String[] patternsToMatch = patternsToMatchList.split(";");
      forloop:
        for (String patternToMatch : patternsToMatch) {
          boolean patternmatchresult = true;
          if(patternmatchresult == true) {
            Pattern regexp = Pattern.compile(".*" + patternToMatch + ".*");
            Matcher matcher = regexp.matcher("");
            LineNumberReader lineReader = null;
            try {
              lineReader = new LineNumberReader( new FileReader(f) );
              String line = null;
              boolean resultinlinewhileloop = false;
                while ((line = lineReader.readLine()) != null && resultinlinewhileloop) {
                  matcher.reset( line ); //reset the input
                  log.debug(testcase + " - Partner " + partnerName + ", line: " + line);
                  if ( matcher.find() ) {
                    patternsMatched++;
                    log.debug(testcase + " - Partner " + partnerName + ", match found in file for pattern: " + patternToMatch);
                    resultinlinewhileloop = true;
                    continue forloop; //continue with next pattern to match in file, doesn't seem to work
                  }
                }
                if(!resultinlinewhileloop) {
                  Failure=true;
                  patternmatchresult = false;
                  FailureMessage=testcase + ";" + partnerName + ";" + partnerReceiveMessageFullPath + ";NOPATTERNMATCH;" + patternToMatch + "|" +  FailureMessage;
                  log.error(testcase + " - Partner " + partnerName + ", pattern could not be matched against the file! " + patternToMatch);
                  break;
                }
            }
            catch (FileNotFoundException ex) {
              ex.printStackTrace();
            }
            catch (IOException ex){
              ex.printStackTrace();
            }
            finally {
              try {
                if (lineReader!= null) lineReader.close();
              }
              catch (IOException ex) {
                ex.printStackTrace();
              }
            }
          }
      }
      log.debug("patternsMatched:" + patternsMatched + ", patternsToMatch:" + patternsToMatch + ", length:" + patternsToMatch.length);
      //check if all patterns could be matched agains the file content
      if (patternsMatched == patternsToMatch.length) {
          log.info(testcase + " - Partner " + partnerName + ", all paterns could be matched against file. Test completed successfully!");
      }
      else {
        Failure=true;
        log.error(testcase + " - Partner " + partnerName + ", some patterns could not be matched against the file! " + patternsToMatchList.toString());
      }
    }
  }
  else {
    log.error(testcase + " - FILE NOT FOUND! Partner " + partnerName + " should have received message with full path " + partnerReceiveMessageFullPath);
    Failure=true;
    FailureMessage=testcase + ";" + partnerName + ";FILENOTFOUND;" + partnerReceiveMessageFullPath + "|" + FailureMessage;
  }
}

In our own project, we wanted to check different partners for every test case. Which resulted in this BeanShell.

We run our JMeter jmx files using Ant. This build file shows how I start our JMeter and create a nice looking report web page of it. I have some code to make some necessary transformations depending if I run the JMeters on our Windows or Linux machines.

<?xml version="1.0" encoding="UTF-8" ?> 
<project basedir="." default="dist_receive" name="Run JMeters to check files received">
  
  <taskdef resource="net/sf/antcontrib/antcontrib.properties"/>
  <property name="windows.local.basepath.location" value="R:" />
  <property name="linux.local.basepath.location" value="/home/user/" />
  
  <condition property="local.basepath.location"
    value="${windows.local.basepath.location}"
    else="${linux.local.basepath.location}">
    <os family="windows" />
  </condition>
  
  <condition property="target.os"
    value="windows"
    else="linux">
    <os family="windows" />
  </condition>  
  
  <property name="test.dir.fullpath" value="${local.basepath.location}/project/test"/>
  <property name="tools.dir.fullpath" value="${local.basepath.location}/tools"/>
  <property name="jmeter.dir.fullpath" value="${tools.dir.fullpath}/jakarta-jmeter-2.3.2"/>
  <property name="jmeter.install.dir.fullpath" value="${jmeter.dir.fullpath}"/>
  <property name="ant.dir.fullpath" value="${tools.dir.fullpath}/apache-ant-1.7.1"/>
  <property name="java.class.path" value="${tools.dir.fullpath}/jdk1.6.0_10/"/>
  <property name="ant.install.dir.fullpath" value="${tools.dir.fullpath}/apache-ant-1.7.1/bin"/>
  
  <property file="${jmeter.install.dir.fullpath}/bin/jmeter.properties"/>
  <property name="testcase.basepath" value="${test.dir.fullpath}/test_data"/>
  <property name="jmeter.result.file.basepath" value="${test.dir.fullpath}/test_results/"/>
  <property name="jmeter.result.filename" value="result"/>
  <property name="jmeter.result.extension" value=".xml"/>
  <property name="jmeter.report.filename" value="report"/>
  <property name="jmeter.report.extension" value=".html"/>
  <property name="jmeter.receive.filename" value="receive"/>
  
  <property name="jmeter.receive.result.file.fullpath" value="${jmeter.result.file.basepath}${jmeter.result.filename}_${jmeter.receive.filename}${jmeter.result.extension}"/>
  <property name="jmeter.receive.report.file.fullpath" value="${jmeter.result.file.basepath}${jmeter.report.filename}_${jmeter.receive.filename}${jmeter.report.extension}"/>
  
  <property name="testcase.receive.extension" value="*_receive.jmx"/>
  
  <taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask">
    <classpath>
      <pathelement location="${jmeter.dir.fullpath}/extras/ant-jmeter-1.0.9.jar"/>
    </classpath>
  </taskdef>
  
  <tstamp>
      <format property="current.date.time" pattern="MM/dd/yyyy hh:mm"/>
  </tstamp>
  <tstamp>
      <format property="xml.current.date" pattern="yyyy-MM-dd"/>
  </tstamp>
  
    
  <target name="dist_receive" description="run jmeter(s) to check received messages">
  
    <property name="test.cases.to.run.basepath" value="${testcase.basepath}/" />
    <property name="test.case.to.perform" value="" />  <!--if the props file does not contain a 'test.name' => the variable needs to be known but no processing should take place. Properties are immutable, if already set by prop file=> not overridden here! -->
    
    <delete file="${jmeter.receive.result.file.fullpath}" />
    <delete file="${jmeter.receive.report.file.fullpath}" />
  
    <if>
      <equals arg1="${test.case.to.perform}" arg2="" />
      <then>
        <echo message="All send cases in dir ${test.cases.to.run.basepath} will be run" />
        <antcall target="run_jmeters">
          <param name="loadtests.basepath" value="${test.cases.to.run.basepath}"/>
          <param name="loadtests.extension" value="${testcase.receive.extension}"/>
          <param name="jmeter.result.file.fullpath" value="${jmeter.receive.result.file.fullpath}"/>
        </antcall>
      </then>
      <else>
        <echo message="Only one testcase is run ${test.case.to.perform}" />
        <antcall target="run_jmeter">
          <param name="jmx.file.fullpath" value="${test.cases.to.run.basepath}/${test.case.to.perform}"/>
          <param name="jmeter.result.file.fullpath" value="${jmeter.receive.result.file.fullpath}"/>
        </antcall>
      </else>
    </if>
    
    <antcall target="make_report">
      <param name="jmeter.result.file.fullpath" value="${jmeter.receive.result.file.fullpath}"/>
      <param name="jmeter.report.file.fullpath" value="${jmeter.receive.report.file.fullpath}"/>
    </antcall>
  </target>
  
  <target name="run_jmeter" description="run jmeter">
    <echo message="Starting to run test: ${jmx.file.fullpath}" /> <!-- ant call param -->
    <copy file="${jmx.file.fullpath}" tofile="${jmx.file.fullpath}.${target.os}" overwrite="true"/>    
    <if>
      <equals arg1="${target.os}" arg2="linux" />
      <then>
        <echo message=" Replacing paths in jmeter data to match the target file system ${target.os}"/>
        <replace file="${jmx.file.fullpath}.${target.os}" token="${windows.local.basepath.location}" value="${linux.local.basepath.location}"/>
        <echo message=" Replacing path delimiters \ into linux delimiter /"/>
        <replace file="${jmx.file.fullpath}.${target.os}" token="\" value="/"/>  
      </then>
    </if>
    <jmeter jmeterhome="${jmeter.install.dir.fullpath}" resultlog="${jmeter.result.file.fullpath}" testplan="${jmx.file.fullpath}.${target.os}" />
    
    <delete file="${jmx.file.fullpath}.${target.os}" />
  </target>
  
  <target name="run_jmeters" description="run all jmeters in dir">
    <echo message="Starting to run test: ${loadtests.basepath}" /> <!-- ant call param -->
    <copy todir="${loadtests.basepath}" overwrite="true">
      <fileset dir="${loadtests.basepath}">
        <include name="${loadtests.extension}"/>
      </fileset>
      <globmapper from="*" to="*.${target.os}"/>
    </copy>
    <if>
      <equals arg1="${target.os}" arg2="linux" />
      <then>
        <echo message=" Replacing paths in jmeter data to match the target file system ${target.os}"/>  
        <replace dir="${loadtests.basepath}" token="${windows.local.basepath.location}" value="${linux.local.basepath.location}">
          <include name="*.${target.os}"/>
        </replace>
        <echo message=" Replacing path delimiters \ into linux delimiter /"/>
        <replace dir="${loadtests.basepath}" value="/">
          <include name="*.${target.os}"/>
          <replacetoken>\</replacetoken>
        </replace>
      </then>
    </if>
    <jmeter jmeterhome="${jmeter.install.dir.fullpath}" resultlog="${jmeter.result.file.fullpath}">
      <testplans dir="${loadtests.basepath}" includes="${loadtests.extension}.${target.os}"/>
    </jmeter>
    
    <delete>
      <fileset dir="${loadtests.basepath}" includes="*.${target.os}"/>
    </delete>
  </target>
  
  
  <target name="make_report" description="make jmeter reports">
    <echo message="Creating test report from: ${jmeter.result.file.fullpath} in: ${jmeter.report.file.fullpath}" />
    <xslt in="${jmeter.result.file.fullpath}" out="${jmeter.report.file.fullpath}" style="${jmeter.install.dir.fullpath}/extras/jmeter-results-detail-report_21.xsl">
      <param name="date" expression="${current.date.time}"/>
      <param name="test_name" expression="CBS3"/>
    </xslt>
  </target>
    
</project>

All files can be downloaded at once using this link.

Friday, April 3, 2009

Excel resource exporter – special characters (Unicode) convertor

Excel_2007On many project I worked, we often keep some program resources in an Excel sheet (labels, translations, static data,  etc.). Excel makes it easy to get an overview of all language translations next to each other and we can easily export this data to text files to be used by our applications.

In one sheet, we also wanted to convert all special characters (e.g. é, ü, etc) to their special notation that will be threaded correctly into our Java application (e.g. è becomes &#232;). And we even used some labels inside JavaScript popup where we wanted a Unicode representation for special characters and some escaped characters (e.g. é becomes \u00E9).

To export all this data, but also make the Excel sheet readable, I implemented some VBS to convert and export the data of the sheet. I even added some extras, for example to make an archive of the exported data, to easily select a folder with a navigator, make the complete sheet readable (convert special characters) etc. All settings can easily be configured.

I suppose parts of the VBS and the look-and-feel can be reused in some other projects, the Excel sheet can be downloaded here.