Sunday, April 12, 2009

My “Firefox add-ons” selection

Firefox has one big benefit: it’s support for Add-ons / Plug- ins / Themes / Extensions. Without this benefit, I simply wouldn’t use Firefox, but only Google Chrome.

I list below, the Add-ons I’m using mostly:

  • AI Roboform toolbar: best way to safely keep track of your website logins, passwords, forms. Even better with it’s nice and easy synchronization to Windows Mobile PDA. the nice toolbar allows you to navigate to pages you use most and immediately login, with just 2 or 3 characters to type!
  • All-in-one Gestures extension: easily navigate using your mouse (navigate using mouse movements and special mouse click combinations)
  • Cooliris picture/movie viewer: great way to watch pictures from many sites (facebook, flickr, google images, youtube, …). The new beta 1.10 version has even better support for facebook.
  • Delicious bookmarks: keep your bookmarks online, so you can easily find back what you liked before, everywhere, anytime. This plug-in has a nice sidebar and easy integration.
  • Facebook toolbar: some extra’s for facebook
  • Fetch Text URL: select some non clickable or miss-spelled url and easily open the url in a new tab.
  • Flashgot: integration of Flashget (download manager) in Firefox.
  • Firebug: required add-on for every web developer. Nice JavaScript debugging possibilities.
  • Greasemonkey: framework to easily install scripts to customize your navigation in specific websites. Thousands of scripts are available for many well known sites. Some of the scripts I prefer:
  • IE Tab: some sites still require Internet Explorer. With this plug-in you can combine Internet Explorer within Firefox.
  • NextPlease: add some keyboard and mouse shortcuts to easily navigate through sites with ‘next’ and ‘previous’ kind of links.
  • SendTo: Send selected text to a text file, together with url and timestamp. SendTo v0.4 is not compatible with Firefox 3.0, but I just changed the compatibility settings in the plug-in and it still works very well now. I use this plugin a lot to always keep a text file with some information on applications I download. I download a lot of software and want to be able to easily retrieve what I’ve downloaded before or get some basic information on the tool. I made an UltraEdit macro to copy the text from the sendto file into a new file, this way I can even retrieve all I need even faster in the format I prefer. All files (SendTo v0.4 for FF 3, batch file to start ultra edit with macro, UltraEdit macro) can be downloaded here.
  • Tab Mix Plus: add extra options and extra functionality for the tabs in Firefox. For example duplicate current tab, open tab on mouse hover, show tab close button on mouse hover, etc.
  • WebDeveloper: required add-on for every web developer. See and change many site settings, form settings, CSS settings, javascripts etc.
  • Vista-aero theme: just a nice look and feel, copied from Internet Explorer.

Once Google Chrome will support all this functionality I’ll dump Firefox and will finally get my 400MB stolen memory back.

UPDATE 05/10/2009: added links to my own Greasemonkey scripts

UPDATE 01/06/2010: added AutoPagerize Greasemonkey link

JMeter to check files and file content

Recently, we needed a JMeter which could perform some tests to make sure files were created on the file system,  matching a specific file path and filename, and some text had to match in the file itself. These files were created by a simulator, but to test our project, we wanted to automatically check if the simulator received all required files.

Apparently, JMeter is not often used for such a checks on the file system, but we preferred JMeter so we could combine the result reports nicely with the reports of the sending of the files (performed by another JMeter).

We succeeded by using BeanShell scripts in the JMeter with a BeanShell Sampler and a BeanShell Assertion. I created a template JMeter jmx and replaced some tags (@…@) in this template for each of our test cases. An example of a filled in JMeter jmx file based on this template is available here. We transform the template to the specific test case jmx file using Excel macros for easy configuration management.

In the JMeter we always linked a CheckFileSampler.bsh and CheckFileAssertion.bsh scripts to perform the required checks. Unlimited properties can be assigned in the JMeter jmx and these properties can very easily be retrieved inside the BeanShell script by using:

value = vars.get("<variable name>");


In the BeanShell script, one can just use standard Java code. I made a simplified version of the code, which is less related to our project. In this code, I build up the path of the file which is expected to exist. The file path is build up based on the properties set in the jmx file. I check if the file exists. If it could be found, I match some patterns against the content of the file. If all goes well, the test case succeeds, else it fails.



import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.io.*;
import java.util.regex.*;
Failure=false;
FailureMessage="";
String testcase = vars.get("testcase");
String simulatorReceiveMessageBasePath = vars.get("simulatorReceiveMessageBasePath");
String partnerName = vars.get("partnerNameProperty");
//should the file exist (be received) or not for this partner? 
String partnerReceiveMessageRequired = vars.get(partnerName + "_partnerRequiredReceiveProperty");
//the following pattern should match within the file (if it exists), semicolon (;) separeted list
String patternsToMatchList = vars.get("patternsToMatchList");
//within the basepath, the simulator creates a new folder on every start of the simulator, we wanted to retrieve the newest folder
//retrieve newest folder from partner
File basefolder = new File(simulatorReceiveMessageBasePath);
if (!basefolder.isDirectory()) {
  Failure = true;
  log.error(testcase + " - Partner " + partnerName  + " incorrect base folder: " + simulatorReceiveMessageBasePath);
  FailureMessage = testcase + ";" + partnerName  + ";INCORRECT_BASE_FOLDER;" + simulatorReceiveMessageBasePath + "|" + FailureMessage;
  break;
}  
File[] foldersinbasefolder = basefolder.listFiles();
Comparator filecomp = new Comparator() {
      public int compare(Object o1, Object o2) {
        return new Long(((File)o2).lastModified()).compareTo
             (new Long(((File) o1).lastModified()));
      }
    };  
Arrays.sort( foldersinbasefolder, filecomp);    
if (foldersinbasefolder == null || foldersinbasefolder.length == 0) {
  Failure = true;
  log.error(testcase + " - Partner " + partnerName  + " no subfolder could be found in the base folder: " + simulatorReceiveMessageBasePath);
  FailureMessage = testcase + ";" + partnerName  + ";NOSUBFOLDERSINBASEFOLDER;" + simulatorReceiveMessageBasePath + "|" + FailureMessage;
  break;
}    
String partnerSimulatorFolder  = foldersinbasefolder[0].getCanonicalPath();
String partnerReceiveMessageFullPath = partnerSimulatorFolder + "\\" + testcase + "_message.xml";
if (partnerReceiveMessageRequired != null && partnerReceiveMessageRequired.equals("1"))  {
      
  //partner should have received the message
  log.info(testcase + " - Partner " + partnerName + " should have received message with full path " + partnerReceiveMessageFullPath);
  File f = new File(partnerReceiveMessageFullPath);
  if (f.exists() && f.isFile() && f.length() > 0) {
    log.info(testcase + " - Partner " + partnerName + " has received message with full path " + partnerReceiveMessageFullPath + " size: " + f.length() + "bytes");
      
    int patternsMatched = 0;
    if (patternsToMatchList != null && patternsToMatchList != "") {
      
      //check if the pattern can be matched with the file
      String[] patternsToMatch = patternsToMatchList.split(";");
      forloop:
        for (String patternToMatch : patternsToMatch) {
          boolean patternmatchresult = true;
          if(patternmatchresult == true) {
            Pattern regexp = Pattern.compile(".*" + patternToMatch + ".*");
            Matcher matcher = regexp.matcher("");
            LineNumberReader lineReader = null;
            try {
              lineReader = new LineNumberReader( new FileReader(f) );
              String line = null;
              boolean resultinlinewhileloop = false;
                while ((line = lineReader.readLine()) != null && resultinlinewhileloop) {
                  matcher.reset( line ); //reset the input
                  log.debug(testcase + " - Partner " + partnerName + ", line: " + line);
                  if ( matcher.find() ) {
                    patternsMatched++;
                    log.debug(testcase + " - Partner " + partnerName + ", match found in file for pattern: " + patternToMatch);
                    resultinlinewhileloop = true;
                    continue forloop; //continue with next pattern to match in file, doesn't seem to work
                  }
                }
                if(!resultinlinewhileloop) {
                  Failure=true;
                  patternmatchresult = false;
                  FailureMessage=testcase + ";" + partnerName + ";" + partnerReceiveMessageFullPath + ";NOPATTERNMATCH;" + patternToMatch + "|" +  FailureMessage;
                  log.error(testcase + " - Partner " + partnerName + ", pattern could not be matched against the file! " + patternToMatch);
                  break;
                }
            }
            catch (FileNotFoundException ex) {
              ex.printStackTrace();
            }
            catch (IOException ex){
              ex.printStackTrace();
            }
            finally {
              try {
                if (lineReader!= null) lineReader.close();
              }
              catch (IOException ex) {
                ex.printStackTrace();
              }
            }
          }
      }
      log.debug("patternsMatched:" + patternsMatched + ", patternsToMatch:" + patternsToMatch + ", length:" + patternsToMatch.length);
      //check if all patterns could be matched agains the file content
      if (patternsMatched == patternsToMatch.length) {
          log.info(testcase + " - Partner " + partnerName + ", all paterns could be matched against file. Test completed successfully!");
      }
      else {
        Failure=true;
        log.error(testcase + " - Partner " + partnerName + ", some patterns could not be matched against the file! " + patternsToMatchList.toString());
      }
    }
  }
  else {
    log.error(testcase + " - FILE NOT FOUND! Partner " + partnerName + " should have received message with full path " + partnerReceiveMessageFullPath);
    Failure=true;
    FailureMessage=testcase + ";" + partnerName + ";FILENOTFOUND;" + partnerReceiveMessageFullPath + "|" + FailureMessage;
  }
}


In our own project, we wanted to check different partners for every test case. Which resulted in this BeanShell.



We run our JMeter jmx files using Ant. This build file shows how I start our JMeter and create a nice looking report web page of it. I have some code to make some necessary transformations depending if I run the JMeters on our Windows or Linux machines.



<?xml version="1.0" encoding="UTF-8" ?> 
<project basedir="." default="dist_receive" name="Run JMeters to check files received">
  
  <taskdef resource="net/sf/antcontrib/antcontrib.properties"/>
  <property name="windows.local.basepath.location" value="R:" />
  <property name="linux.local.basepath.location" value="/home/user/" />
  
  <condition property="local.basepath.location"
    value="${windows.local.basepath.location}"
    else="${linux.local.basepath.location}">
    <os family="windows" />
  </condition>
  
  <condition property="target.os"
    value="windows"
    else="linux">
    <os family="windows" />
  </condition>  
  
  <property name="test.dir.fullpath" value="${local.basepath.location}/project/test"/>
  <property name="tools.dir.fullpath" value="${local.basepath.location}/tools"/>
  <property name="jmeter.dir.fullpath" value="${tools.dir.fullpath}/jakarta-jmeter-2.3.2"/>
  <property name="jmeter.install.dir.fullpath" value="${jmeter.dir.fullpath}"/>
  <property name="ant.dir.fullpath" value="${tools.dir.fullpath}/apache-ant-1.7.1"/>
  <property name="java.class.path" value="${tools.dir.fullpath}/jdk1.6.0_10/"/>
  <property name="ant.install.dir.fullpath" value="${tools.dir.fullpath}/apache-ant-1.7.1/bin"/>
  
  <property file="${jmeter.install.dir.fullpath}/bin/jmeter.properties"/>
  <property name="testcase.basepath" value="${test.dir.fullpath}/test_data"/>
  <property name="jmeter.result.file.basepath" value="${test.dir.fullpath}/test_results/"/>
  <property name="jmeter.result.filename" value="result"/>
  <property name="jmeter.result.extension" value=".xml"/>
  <property name="jmeter.report.filename" value="report"/>
  <property name="jmeter.report.extension" value=".html"/>
  <property name="jmeter.receive.filename" value="receive"/>
  
  <property name="jmeter.receive.result.file.fullpath" value="${jmeter.result.file.basepath}${jmeter.result.filename}_${jmeter.receive.filename}${jmeter.result.extension}"/>
  <property name="jmeter.receive.report.file.fullpath" value="${jmeter.result.file.basepath}${jmeter.report.filename}_${jmeter.receive.filename}${jmeter.report.extension}"/>
  
  <property name="testcase.receive.extension" value="*_receive.jmx"/>
  
  <taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask">
    <classpath>
      <pathelement location="${jmeter.dir.fullpath}/extras/ant-jmeter-1.0.9.jar"/>
    </classpath>
  </taskdef>
  
  <tstamp>
      <format property="current.date.time" pattern="MM/dd/yyyy hh:mm"/>
  </tstamp>
  <tstamp>
      <format property="xml.current.date" pattern="yyyy-MM-dd"/>
  </tstamp>
  
    
  <target name="dist_receive" description="run jmeter(s) to check received messages">
  
    <property name="test.cases.to.run.basepath" value="${testcase.basepath}/" />
    <property name="test.case.to.perform" value="" />  <!--if the props file does not contain a 'test.name' => the variable needs to be known but no processing should take place. Properties are immutable, if already set by prop file=> not overridden here! -->
    
    <delete file="${jmeter.receive.result.file.fullpath}" />
    <delete file="${jmeter.receive.report.file.fullpath}" />
  
    <if>
      <equals arg1="${test.case.to.perform}" arg2="" />
      <then>
        <echo message="All send cases in dir ${test.cases.to.run.basepath} will be run" />
        <antcall target="run_jmeters">
          <param name="loadtests.basepath" value="${test.cases.to.run.basepath}"/>
          <param name="loadtests.extension" value="${testcase.receive.extension}"/>
          <param name="jmeter.result.file.fullpath" value="${jmeter.receive.result.file.fullpath}"/>
        </antcall>
      </then>
      <else>
        <echo message="Only one testcase is run ${test.case.to.perform}" />
        <antcall target="run_jmeter">
          <param name="jmx.file.fullpath" value="${test.cases.to.run.basepath}/${test.case.to.perform}"/>
          <param name="jmeter.result.file.fullpath" value="${jmeter.receive.result.file.fullpath}"/>
        </antcall>
      </else>
    </if>
    
    <antcall target="make_report">
      <param name="jmeter.result.file.fullpath" value="${jmeter.receive.result.file.fullpath}"/>
      <param name="jmeter.report.file.fullpath" value="${jmeter.receive.report.file.fullpath}"/>
    </antcall>
  </target>
  
  <target name="run_jmeter" description="run jmeter">
    <echo message="Starting to run test: ${jmx.file.fullpath}" /> <!-- ant call param -->
    <copy file="${jmx.file.fullpath}" tofile="${jmx.file.fullpath}.${target.os}" overwrite="true"/>    
    <if>
      <equals arg1="${target.os}" arg2="linux" />
      <then>
        <echo message=" Replacing paths in jmeter data to match the target file system ${target.os}"/>
        <replace file="${jmx.file.fullpath}.${target.os}" token="${windows.local.basepath.location}" value="${linux.local.basepath.location}"/>
        <echo message=" Replacing path delimiters \ into linux delimiter /"/>
        <replace file="${jmx.file.fullpath}.${target.os}" token="\" value="/"/>  
      </then>
    </if>
    <jmeter jmeterhome="${jmeter.install.dir.fullpath}" resultlog="${jmeter.result.file.fullpath}" testplan="${jmx.file.fullpath}.${target.os}" />
    
    <delete file="${jmx.file.fullpath}.${target.os}" />
  </target>
  
  <target name="run_jmeters" description="run all jmeters in dir">
    <echo message="Starting to run test: ${loadtests.basepath}" /> <!-- ant call param -->
    <copy todir="${loadtests.basepath}" overwrite="true">
      <fileset dir="${loadtests.basepath}">
        <include name="${loadtests.extension}"/>
      </fileset>
      <globmapper from="*" to="*.${target.os}"/>
    </copy>
    <if>
      <equals arg1="${target.os}" arg2="linux" />
      <then>
        <echo message=" Replacing paths in jmeter data to match the target file system ${target.os}"/>  
        <replace dir="${loadtests.basepath}" token="${windows.local.basepath.location}" value="${linux.local.basepath.location}">
          <include name="*.${target.os}"/>
        </replace>
        <echo message=" Replacing path delimiters \ into linux delimiter /"/>
        <replace dir="${loadtests.basepath}" value="/">
          <include name="*.${target.os}"/>
          <replacetoken>\</replacetoken>
        </replace>
      </then>
    </if>
    <jmeter jmeterhome="${jmeter.install.dir.fullpath}" resultlog="${jmeter.result.file.fullpath}">
      <testplans dir="${loadtests.basepath}" includes="${loadtests.extension}.${target.os}"/>
    </jmeter>
    
    <delete>
      <fileset dir="${loadtests.basepath}" includes="*.${target.os}"/>
    </delete>
  </target>
  
  
  <target name="make_report" description="make jmeter reports">
    <echo message="Creating test report from: ${jmeter.result.file.fullpath} in: ${jmeter.report.file.fullpath}" />
    <xslt in="${jmeter.result.file.fullpath}" out="${jmeter.report.file.fullpath}" style="${jmeter.install.dir.fullpath}/extras/jmeter-results-detail-report_21.xsl">
      <param name="date" expression="${current.date.time}"/>
      <param name="test_name" expression="CBS3"/>
    </xslt>
  </target>
    
</project>


All files can be downloaded at once using this link.


UPDATE 10/02/2010: new link for full package.

Friday, April 3, 2009

Excel resource exporter – special characters (Unicode) convertor

Excel_2007On many project I worked, we often keep some program resources in an Excel sheet (labels, translations, static data,  etc.). Excel makes it easy to get an overview of all language translations next to each other and we can easily export this data to text files to be used by our applications.

In one sheet, we also wanted to convert all special characters (e.g. é, ü, etc) to their special notation that will be threaded correctly into our Java application (e.g. è becomes &#232;). And we even used some labels inside JavaScript popup where we wanted a Unicode representation for special characters and some escaped characters (e.g. é becomes \u00E9).

To export all this data, but also make the Excel sheet readable, I implemented some VBS to convert and export the data of the sheet. I even added some extras, for example to make an archive of the exported data, to easily select a folder with a navigator, make the complete sheet readable (convert special characters) etc. All settings can easily be configured.

I suppose parts of the VBS and the look-and-feel can be reused in some other projects, the Excel sheet can be downloaded here.

Thursday, April 2, 2009

House mortgage calculator – Excel template (Flemish)

Since we recently bought a new house, I spent some time to be able to calculate the best construction for our house credit / loan / mortgage. I tried to list all costs in an Excel sheet, combined with our earnings, to get a clear global picture on what we will have to pay. Based on these calculations, I extracted an Excel template that could be used by anyone. The costs taken into account are quite specific for Belgium (Flanders) and for our case but I suppose the sheet could be used as a start for others (let me know if this is the case)…

When using this template, some data will have to be filled in, before the calculated results will be meaningful. The fields that can be filled in/changed/selected manually, are marked in the following style: 

fieldtofill

The most important fields to fill in at start are: price, cadastral income, savings, average pay, average outlay, interest rate of the banks, etc within the ‘kosten’ sheet. Within the ‘interesten’ sheet only at the top some fields should be filled up to denote how you would like to calculate a split loan (to benefit most of the fiscal revenue). Again the same format style is used to denote these fields should be filled in. The other fields are locked on the sheet (although they can easily be unlocked without a password).

Once the basic numbers are filled in, one can start to play with the list boxes to easily compare different constructions (split loan, fixed capital payment, years, banks, etc). One can now change the control to show how much will have to be paid and gained in the selected year. This way, you can easily check if you will be able to pay all costs and still save some money.yearselectorThe template can be downloaded here.



The site rentevoet.be has also made a nice excel to calculate different loan payments, besides some other calculators.




Some interesting links I used to make all calculations:


Update 08/10/09: extra info link

Update 05/01/10: new template download link

Update 03/12/10: link to renteopdevoet.be