Muhimbi

Posted at: 3:11 PM on 12 March 2010 by Muhimbi

In part of 2 of our series about porting a SharePoint 2007 based WSPBuilder project to SharePoint 2010 we discuss the changes made to our Visual Studio 2008 based project to support both versions of SharePoint

Please note that this article is based on our experiences with the beta version of SharePoint 2010. Some of the issues we have identified may have been resolved in the final release.  

The following posts are part of this series:

• Part 1 – Introduction / Problems installing the existing 2007 version on SP2010
• Part 2 – Reconfiguring the Visual Studio Solution (This post)
• Part 3 – Programmatic / visual changes (Available soon)
• Part 4 – Updating deployment scripts (Available soon)
   
  

WSP Builder

On January 13 a new ‘2010’ compatible version of WSPBuilder was released. I seriously doubt this new version is required in order to build SharePoint 2010 compatible solutions, but we upgraded to it nevertheless.

Note that if you want to use WSPBuilder to build a hybrid SharePoint 2007 / 2010 solution or a solution that just targets SharePoint 2007 then you must perform the build on a machine that runs SharePoint 2007. The latest version of WSPBuilder checks which version of SharePoint is installed on the build machine and adds SharePoint 2010 specific elements to the generated WSP file, causing deployments to fail in SharePoint 2007 environments.

Unfortunately it does not seem to be possible to modify this behaviour using a command line switch.

Visual Studio Project structure

Depending on the complexity of your project, and on the need to add any SharePoint 2010 specific functionality, you may not need to make any changes to your project structure.

However, in our case we decided to make the following changes:

1. We renamed the 12 folder to SPHive. From a functional perspective there is no need to make this change, however it provides our developers with a more consistent experience as, depending on the platform they are targeting, the files may go to either the 12 or 14 hive.
    
2. A separate shadow SPHive_2010 folder was created to store SharePoint 2010 specific copies of files that will break compatibility when deployed to SharePoint 2007. For example element files that contain Custom Actions that target the SharePoint 2010 specific Ribbon will prevent a WSP file from deploying to SharePoint 2007. Application pages that contain the DynamicMasterPageFile attribute will break compatibility as well.
    

When a build is carried out the two SPHive folders are merged and 2 separate WSP files are created. For details see the next section.

Post Build event

Although the WSPBuilder Extensions for Visual Studio are great, our projects don’t use them. Instead our main project’s Post Build event invokes either WSPBuilder manually to generate WSP files or carries out a deployment using XCOPY.

As part of this exercise we have made the following changes:

1. Auto detect the installed version of SharePoint.
2. Determine the location of the SharePoint root directory (12 or 14 Hive).
3. Added support for generating both SharePoint 2007 and 2010 WSP files when doing a release build.
4. Added the ability to carry out an XCOPY deployment of a merged version of the SPHive and SPHive_2010 folders..
    

A simplified copy of our PDF Converter’s post build event is included below. As all our development servers run the 64 bit version of Win2K8 or Win2K8R2 this script may not work on Win2K3 or 32 bit installations. Some long lines, especially those invoking WSPBuilder, have been wrapped and reformatted for readability. When copying this script please make sure that multi-line commands are all placed on a single line.

set useWSPBuilder=false

set gacutil="$(SolutionDir)..\..\SharedBinaries\GACUtil\gacutil.exe"

set wspbuilder="$(SolutionDir)..\..\SharedBinaries\WSPBuilder\wspbuilder.exe"

set SPHive_2010=$(ProjectDir)SPHive_2010

@echo ** Detect which version of SharePoint is installed.

set CommonProgramsFolder=%CommonProgramW6432%

set SharePointRoot=%CommonProgramsFolder%\Microsoft Shared\Web Server Extensions\14

set IsSP2010=true

if exist "%SharePointRoot%\bin\STSADM.EXE" goto endVersionDetection

        set SharePointRoot=%CommonProgramsFolder%\Microsoft Shared\Web Server Extensions\12

        set IsSP2010=false

        if exist "%SharePointRoot%\bin\STSADM.EXE" goto endVersionDetection

        echo ** SharePoint does not appear to be installed on this server.

        goto end

:endVersionDetection

echo Detected SharePoint Root: "%SharePointRoot%"

if $(ConfigurationName)==Debug goto debugMode

        @echo ** Not running in debug mode so enabling WSPBuilder

        set useWSPBuilder=true

:debugMode

@REM Do we want to build using XCopy or WSPBuilder?

if %useWSPBuilder%==false goto useXCopy

    @echo ** Build mode: WSP Builder

    @echo ** Remove files from the WSPBuilder GAC Directory

    del /F /Q "$(ProjectDir)GAC\*.*"

    @echo ** Move dependent DLLs to GAC directory to allow WSPBuilder to package them up

    move /y "$(TargetDir)Muhimbi.SharePoint.Common.dll" "$(ProjectDir)GAC"

    move /y "$(TargetDir)Muhimbi.SharePoint.Diagnostics.dll" "$(ProjectDir)GAC"

    move /y "$(TargetDir)Muhimbi.Licensing.Base.dll" "$(ProjectDir)GAC"

    move /y "$(TargetDir)Muhimbi.Licensing.Validator.dll" "$(ProjectDir)GAC"

    move /y "$(TargetDir)Muhimbi.Licensing.SharePoint.dll" "$(ProjectDir)GAC"

    @echo ** Building WSP for SharePoint 2007

    %wspbuilder% -BuildDDF True -BuildWSP true -ResetWebServer False -Outputpath 

        "$(ProjectDir)Solution" -12Path "$(ProjectDir)SPHive" -GACPath "$(ProjectDir)GAC" 

        -ProjectPath "$(ProjectDir)" -Cleanup True -Excludefiletypes "cmd,cs,scc" 

        -WSPName Muhimbi.PDFConverter.wsp -BuildSafeControls False

    @echo ** Building SPHive for SharePoint 2010

    set SPHive_Temp=$(ProjectDir)_SPHive_Temp

    rm -f -r "%SPHive_Temp%"

    md "%SPHive_Temp%"

    xcopy /E /Y "$(ProjectDir)SPHive" "%SPHive_Temp%"

    xcopy /E /Y "%SPHive_2010%" "%SPHive_Temp%"

    %wspbuilder% -BuildDDF True -BuildWSP true -ResetWebServer False -Outputpath 

        "$(ProjectDir)Solution" -12Path "%SPHive_Temp%" -GACPath "$(ProjectDir)GAC" 

        -ProjectPath "$(ProjectDir)" -Cleanup True -Excludefiletypes "cmd,cs,scc" 

        -WSPName Muhimbi.PDFConverter.SP2010.wsp -BuildSafeControls False

    @echo ** Cleaning up

    rm -f -r "%SPHive_Temp%"

    del /F /Q "$(ProjectDir)Solution\makecab.ddf"

    goto end

:useXCopy

    @echo ** Build mode: XCOPY

    xcopy /E /Y "$(ProjectDir)SPHive" "%SharePointRoot%"

    if %IsSP2010%==false goto skipSP2010Copy

        @echo Copying SharePoint 2010 specific files

        xcopy /E /Y "%SPHive_2010%" "%SharePointRoot%"

    :skipSP2010Copy

    @echo ** Installing GAC assemblies

    %gacutil% /if "$(TargetDir)Muhimbi.SharePoint.Common.dll"

    %gacutil% /if "$(TargetDir)Muhimbi.SharePoint.Diagnostics.dll"

    %gacutil% /if "$(TargetDir)Muhimbi.Licensing.Base.dll"

    %gacutil% /if "$(TargetDir)Muhimbi.Licensing.Validator.dll"

    %gacutil% /if "$(TargetDir)Muhimbi.Licensing.SharePoint.dll"

    @echo ** Installing $(OutDir)$(TargetFileName) into the GAC...

    %gacutil% /if "$(TargetPath)"

    echo ** Recycling App Pools

    cscript //NoLogo "$(SolutionDir)\RecycleAppPools.vbs"

:end

@echo ** rebuilding sitemaps and translations

"%SharePointRoot%\bin\STSADM.EXE" -o copyappbincontent

The vbscript file that we use to recycle the application pools at the end of the build process was not working on our SharePoint 2010 development machine for some reason. We are not sure if this is related to the fact that we migrated to Win2K8R2 (from Win2K8) as part of the SharePoint 2010 deployment process or if it is because the script was always broken. Basically it fell over if one of the Application Pools was not started.

The new script is now as follows:

Set oWebAdmin = GetObject("winmgmts:root\WebAdministration")

Set oAppPools = oWebAdmin.InstancesOf("ApplicationPool")

For Each oAppPool In oAppPools

    WScript.Echo "Recycling application pool: " & oAppPool.Name

    '** Only recycle pools that are currently started

    if oAppPool.GetState = 1 then

        oAppPool.Recycle    

    end if

Next

Continue to Part 3 – Programmatic / visual changes (Available soon).

.

Labels: Articles, News, PDF Converter, Products, SP2010, Workflow

Posted at: 5:01 PM on 11 March 2010 by Muhimbi

When we decided to make our popular PDF Converter for SharePoint compatible with SharePoint 2010, we had no idea what we were in for. Will it be a nightmare, will it just work, will we need to throw everything away?…. we simply didn’t know. Fortunately SharePoint 2010 is much like SharePoint 2007 and as a result we released the SharePoint 2010 compatible version earlier today.

As there is very little available information or guidance on this topic, we would like to share our experience with migrating SharePoint 2007 based WSPBuilder projects to SharePoint 2010 while maintaining a single code base that supports both environments.

This post, the first one in the series, describes what happened when we tried to install a SharePoint 2007 based solution on a machine running SharePoint 2010.

The following posts are part of this series:

• Part 1 – Problems installing the existing 2007 version on SP2010 (this post)
• Part 2 – Reconfiguring the Visual Studio Solution
• Part 3 – Programmatic / visual changes (Available soon)
• Part 4 – Updating deployment scripts (Available soon)

Please note that this article is based on our experiences with the beta version of SharePoint 2010. Some of the issues we have identified may have been resolved in the final release.

Our goal

When we started the planning phase for the SharePoint 2010 migration, we decided that ideally we would end up with a single code base that can be used to build a single WSP file that is compatible with both SharePoint 2007 and 2010. To make the product easy to maintain we only want a single Visual Studio Solution and as little code duplication as possible. At the same time we wanted to leverage SharePoint 2010’s new facilities such as the Ribbon without breaking compatibility.

Although we had to give up our single WSP file dream, more on that later, we managed to achieve our other goals.

Installing a SharePoint 2007 based solution on SharePoint 2010

When we embarked on our migration project we briefly thought: what if it just works? That would have been great, but unfortunately we stumbled at the first hurdle, the installer.

The installer for the SharePoint WFE part of the solution is a simple cmd script that invokes stsadm directly. In SharePoint 2010 stsadm lives in a different directory from its SharePoint 2007 counterpart so our dream ended rather abruptly.

Once we had fixed the installer we stumbled onto a rather annoying problem that kept us busy for the better part of 2 days. Whenever we tried to deploy a WSP file (any WSP file it turned out) the process failed without any error messages. However, the following message was logged to the Windows Application Event log: Requested registry access is not allowed.

After consulting many sources, it turned out that if the first attempt to install SharePoint 2010 fails (ours did) then the privileges on the HKLM\SOFTWARE\Microsoft \Shared Tools\Web Server Extensions\14.0\Secure\FarmAdmin registry key are not configured during subsequent successful installations. After giving the Everyone group full control on this key it was finally possible to deploy WSP files. (I realise that giving Everyone access is not best practice, but it solved the problem on our development server).

For details about the changes we had to make to our deployment scripts see part 4 of this series.

Where are my menu options?

Once the wsp file was installed and deployed, the Convert to PDF option became visible in the ECB (file context menu), woohooo! Unfortunately, the link to our Central Administration based configuration screen was missing. Because the Central Administration screens have been restructured, the Locations recognised by SharePoint 2007 have no direct mapping to their SharePoint 2010 equivalents. It is a shame that SharePoint is not automatically remapping those Locations that have a direct equivalent in SharePoint 2010. After all, the External Service Connections Location is available in both versions.
 

Link to the PDF Converter Settings screen in SharePoint 2007

For details about the changes we had to make in order to make the menus work, see part 3 of this series.

Why does it all look wrong?

The main problems we encountered during this entire exercise were related to the visual fidelity of our Application screens. We take great pride in the appearance of our solutions so we were disappointed to see that, even though we have gone through great lengths to only use those user interface elements and controls that ship with SharePoint, things didn’t look quite right.

The main visual problems we encountered are as follows:

1. The vertical alignment of checkboxes is completely off.
2. There is extra vertical spacing between the various user interface elements.
3. Some elements, particularly buttons, have a different default width and therefore wrap over multiple lines.

 
The following screenshots provide a visual comparison between the SharePoint 2007 and 2010 versions before any changes were made.

  Original SharePoint 2007 interface and the SharePoint 2010 one before making any changes

This concludes the overview of identified problems.

Continue to Part 2 – Reconfiguring the Visual Studio Solution.

.

Labels: Articles, News, PDF Converter, Products, SP2010, Workflow

Posted at: 10:09 AM on by Muhimbi

When we released the Office 2010 compatible version of the Muhimbi PDF Converter for SharePoint we were already planning, and worrying about, a SharePoint 2010 compatible version. Worrying indeed, call it fear of the unknown or fear of change, but we were anticipating a lot of work.

Fortunately we quickly found out that SharePoint 2010 is, surprise surprise, much like SharePoint 2007. If it wasn’t for an annoying SharePoint 2010 deployment bug that took us 2 days to solve, we could have completed all investigations and required changes in 2 or 3 days.

To cut a rather long story short, a beta version of the PDF Converter for SharePoint 2010 is now available. Please contact us if you want to participate in the beta program. Note that the download available on our website is the SharePoint 2007 version, which will not work with SharePoint 2010.

Although the beta is fully functional and integrates nicely with new SharePoint 2010 features such as the Ribbon, the documentation has not yet been updated. Fortunately, much - including the installation process - is the same. The only difference is the location of some of the administrative screens as highlighted in the screenshots below.

Over the next week we will release a number of blog posts describing our experience with porting a SharePoint 2007 WSPBuilder application to SharePoint 2010 and making the same code base work with both versions. If you are interested in this topic then make sure you subscribe to our RSS feed or follow us on Twitter.

Access the PDF Converter Configuration screen via General Application Settings

  The PDF Converter integrates with the new Ribbon bar

  After installing the License Manager, it can be opened from the System Settings screen

The new version is compatible with SharePoint Designer 2010 workflows

.

Labels: Articles, News, PDF Converter, Products, SP2010, Workflow

Posted at: 3:53 PM on 24 February 2010 by Muhimbi

Earlier today we released a new version of our popular PDF Converter for SharePoint. One of the key changes in this version is that it fully supports Office 2010 file formats, including all new formatting features such as Excel 2010 Sparklines.

Full installation details are provided in Appendix – Office 2010 Installation of the Administration Guide, however one detail is so essential that we feel we have to repeat it in this blog post. After all, who reads boring documentation anyway?

The single most important thing to take away from this post is that no matter what CPU architecture you deploy the PDF Converter on, if you plan to use Office 2010 to carry out the conversions then you should always install the 32 bit version of Office 2010.

Even though the Muhimbi PDF Converter for SharePoint is a hybrid 32 / 64 bit application, the 64 bit version of Office will not work in combination with our software. Even if it did, it would provide little to no benefit.

For more details about the differences between the 32 and 64 bit versions of Office, including Microsoft’s recommendation to run the 32-bit version of Office 2010 on 64 bit hardware, read this article on Microsoft’s Office 2010 blog.

.

Labels: Articles, PDF Converter, Products

Posted at: 2:29 PM on by Muhimbi

Ah, those pesky customers of ours, always looking for some niche functionality that is impossible to include in a generic product. However, using the Workflow Power Pack for SharePoint we can achieve almost anything we can think of…..almost.

Previously I described how to configure PDF Security settings from a SharePoint workflow and how to automatically watermark PDF files from a workflow.  This time I’ll show how to add JavaScript to any PDF file to automatically print the current date (the date the PDF was opened) on every page. In essence this adds a print date without modifying the PDF file every day to include the current date.

A quick introduction for those not familiar with the product: The Muhimbi Workflow Power Pack for SharePoint allows custom C# or VB.NET code to be embedded in SharePoint Designer Workflows without the need to resort to complex Visual Studio based workflows, the development of bespoke Workflow Activities or long development cycles.

The solution presented below executes a workflow whenever a PDF file is added or updated. It iterates over all pages and inserts a form field on each page. Some client side JavaScript is then added to the PDF file that iterates over all newly added fields to insert the current date every time the PDF file is opened.

As the code is well documented it is easy to make further changes and customisations, e.g. change the formatting of the date or position of the label. Note that this has only been tested with a recent version of Adobe Acrobat reader. If you use a different PDF viewer your mileage may vary.

 
Create the workflow as follows:

1. Download and install the Muhimbi Workflow Power Pack for SharePoint.
    
2. Download and install the Muhimbi PDF Converter for SharePoint.
   Note that you need version 3.2.0.20 or newer, older versions do not allow JavaScript to be inserted.
    
3. Download this article’s source code.
    
4. We need to be able to access functionality in the Muhimbi.SharePoint.DocumentConverter.PDF and System.Drawing assemblies. Add these references to the relevant Web Application using the Workflow Power Pack Central Administration screens as described in the Administration Guide. Make sure to place each reference on a new line.
     
5. Make sure you have the appropriate privileges to create workflows on a site collection.
    
6. Create a new workflow using SharePoint Designer.
    
7. On the Workflow definition screen associate the workflow with the Shared Documents library, tick the boxes next to both ‘Automatically start….’ options and proceed to the next screen.
    
8. We only want to act on files of type PDF. Although we could have put this validation in the code, in this example we use a workflow condition for it so add a Compare Any Data Source condition and:
    
   a.  Click on the first value followed by the display data binding (fx) button.
   b.  Select Current Item as the Source and select File Type in the Field. Click the OK button to continue.
   d.  Click on the second value and enter pdf.
    
9. Click the Actions button and insert the Execute Custom Code action.
    
10. Optionally click parameter 1 and enter a relative or absolute destination path. Leave the parameter empty to save the modified file on top of the existing PDF file. For details about how paths are handled, see this post and search for the words ‘this url’.
     
11. Insert the C# based code embedded in step #3’s download (also listed below) by clicking this code.
     

    /*********************************************************************************************

                            Muhimbi PDF Converter - JavaScript Watermarking

                  Copyright 2010, Muhimbi Ltd - www.muhimbi.com - All rights reserved

     The following code shows a simple way of adding JavaScript to existing PDF Files. It adds 

     the current date to each page in the document in order to simulate a 'print date' that is

     always up to date without the need to modify the PDF file. The code is automatically executed 

     when the document is opened in the Adobe Acrobat Viewer.

     Error and permission checking as well as other minor features have been omitted for the sake 

     of brevity and clarity.

     Ideally PDF Conversion, applying security and watermarking is executed in the same step, see 

     http://www.muhimbi.com/blog/2010/01/configure-pdf-security-from-sharepoint.html

     This code requires Muhimbi’s PDF Converter and Workflow Power Pack to be installed.

    *********************************************************************************************/

    using System.Drawing;

    using System.IO;

    using Syncfusion.Pdf;

    using Syncfusion.Pdf.Parsing;

    using Syncfusion.Pdf.Graphics;

    using Syncfusion.Pdf.Interactive;

    using Muhimbi.SharePoint.DocumentConverter.PDF;

    SPFile spSourceDocument = MyWorkflow.Item.File;

    string destinationFileName = spSourceDocument.Name;

    string destinationFolderName = MyWorkflow.Parameter1 as string;

    // ** Load the document

    PdfLoadedDocument sourceDocument = new PdfLoadedDocument(spSourceDocument.OpenBinary());

    PdfDocument destinationDocument = new PdfDocument();

    // ** Copy all pages from the source document into the destination document 

    // ** so we can add JavaScript actions.

    destinationDocument.ImportPageRange(sourceDocument, 0, sourceDocument.Pages.Count - 1);

    sourceDocument.Dispose();

    // ** Iterate over all pages and add a form element

    for (int i = 0; i < destinationDocument.Pages.Count; i++)

    {

        PdfPage destinationPage = destinationDocument.Pages[i];

        // ** Create a new field using a unique name

        PdfTextBoxField field = new PdfTextBoxField(destinationPage, "_M_PrintDateField_" + i);

        // ** Center the field

        const int BOX_WIDTH = 200;

        int boxLeft = (int)((destinationPage.Size.Width - BOX_WIDTH) / 2);

        field.Bounds = new RectangleF(boxLeft, 20, BOX_WIDTH, 20);

        // ** Format the field

        PdfFont font = new PdfStandardFont(PdfFontFamily.Helvetica, 12f);

        field.Font = font;

        field.BorderColor = new PdfColor(Color.White);

        field.BackColor = new PdfColor(Color.White);

        field.ReadOnly = true;

        field.TextAlignment = PdfTextAlignment.Center;

        destinationDocument.Form.Fields.Add(field);

    }

    // ** Create a client side script that iterates over all fields and populates the date

    string jscript = @"

            var pages = " + destinationDocument.Pages.Count + @";

            var today = util.printd('dd-mm-yyyy', new Date());

            for(var i=0; i<pages; i++)

            {

                var field = this.getField('_M_PrintDateField_' + i);

                field.value = 'Today is: ' + today;

            }

            ";

    // ** Attach the script to the Document Open event.

    PdfJavaScriptAction jsAction = new PdfJavaScriptAction(jscript);

    destinationDocument.Actions.AfterOpen = jsAction;

    // ** Construct the path and file to write the watermarked PDF file to.

    if (string.IsNullOrEmpty(destinationFolderName) == true)

        destinationFolderName = spSourceDocument.ParentFolder.Url;

    SPFolder destinationFolder = Utility.GetSPFolder(destinationFolderName, MyWorkflow.Web);

    string destinationFilePath = string.Format("{0}/{1}", destinationFolder.Url,

                                               destinationFileName);

    SPWeb destinationWeb = destinationFolder.ParentWeb;

    SPFile spDestinationFile = destinationWeb.GetFile(destinationFilePath);

    // ** If a document library requires manual checkout and the file is not checked out, then 

    // ** check the file out before uploading.

    if (spDestinationFile.Exists && spDestinationFile.Item.ParentList.ForceCheckout &&

        spDestinationFile.CheckOutStatus == SPFile.SPCheckOutStatus.None)

    {

        spDestinationFile.CheckOut();

    }

    // ** Add the file to the site including the meta data

    using (MemoryStream watermarkedFile = new MemoryStream())

    {

        destinationDocument.Save(watermarkedFile);

        spDestinationFile = destinationWeb.Files.Add(destinationFilePath, watermarkedFile,

                                                     spSourceDocument.Item.Properties, true);

    }

    // ** Check the file back in if this script was responsible for checking it out.

    if (spDestinationFile.Item.ParentList.ForceCheckout == true)

    {

        spDestinationFile.CheckIn("Auto check-in after PDF watermarking.");

    }

     
12. Click the Actions button, select Log to History List, click this message and enter File watermarked.
     
13. Close the Workflow Designer.
     
14. Update an existing PDF or add a new PDF file to your library to trigger the workflow and apply the JavaScript.

  
Naturally this is just a simple example. Feel free to play around with the code, change which parameters are passed into the workflow, or add different JavaScript. Note that you may want to add a check to the code to check if the JavaScript / fields have previously been added, otherwise duplicate form fields may be added every time the PDF is updated.

Adobe’s JavaScript for Acrobat reference can be found here.

.

Labels: Articles, PDF Converter, Products, Workflow, WPP

Posted at: 11:50 AM on 22 February 2010 by Muhimbi

Just a quick note to make sure that anyone typing this error message into a search engine will find this post.

Due to a change in the SharePoint 2010 AllDocVersions table it is no longer possible to rebuild indexes as part of a SQL 2008 maintenance plan and keep the indexes on-line at the same time.

The reason behind this is that the MetaInfo field is no longer of type Image. It is now of type tCompressedBinary:varbinary(MAX).

If you get the error listed below then make sure you open the Rebuild Index Task in your SQL Maintenance plan and disable the ‘Keep index online while reindexing’ option.

If you don’t then you’ll get the following error:

Executing the query "ALTER INDEX [AllDocVersions_PK] ON [dbo].[AllDocVe..." failed with the following error: "An online operation cannot be performed for index 'AllDocVersions_PK' because the index contains column 'MetaInfo' of data type text, ntext, image, varchar(max), nvarchar(max), varbinary(max), xml, or large CLR type. For a non-clustered index, the column could be an include column of the index. For a clustered index, the column could be any column of the table. If DROP_EXISTING is used, the column could be part of a new or old index. The operation must be performed offline.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.

If you use one of the default maintenance plans then this error happens before the Backup step. As a result your databases will not be backed up.

.

Labels: Articles, News, SP2010

Posted at: 1:22 PM on 18 February 2010 by Muhimbi

Not too long ago we wrote about how to create a Short URL from a SharePoint workflow using the Muhimbi URL Shortener (MuSH) in combination with our Workflow Power Pack. The response from our customers has been so positive that we decided to ship a Workflow Action with the new version of MuSH.

For those not familiar with the product, the Muhimbi URL Shortener for SharePoint, aka MuSH, can be used to shorten URLs for typical web applications and SharePoint in particular. It integrates tightly with both WSS and MOSS and allows short URLs to be created directly from a list item’s context menu, workflows and web services. For details see the original product announcement.

Creating short URLs from a workflow can be very useful. For example creating a short URL named after data in an InfoPath form or create a short URL for a deeply nested folder. In the example below we create a short URL that always points to the latest entry in the announcement list. Not sure if this is useful, but it illustrates the power of this facility.

Create the workflow as follows:

1. Download and install the Muhimbi URL Shortener for SharePoint.
       
2. Make sure you have the appropriate privileges to create workflows on a site collection.
    
3. Create a new workflow using SharePoint Designer.
    
4. On the Workflow Definition screen associate the workflow with the Announcements list, tick the box next to ‘Automatically start this workflow when a new item is created’ and proceed to the next screen.
    
5. From the Actions Menu select Create Short URL, you may need to click More Actions first.
    
6. The following Workflow Sentence is inserted:
   
     
7. To auto generate the short URL, leave the optional short name empty, but in our case we always want to give it the same name, so enter Announce.
    
8. Click this ID / address, click the Workflow Lookup button and select Current Item as the Source and ID as the field.
    
9. Click Document / Display Form and select Document (when used in a Document Library) or Display Form showing the item’s properties. As we are not dealing with a Document Library, it doesn’t matter what is selected.
    
10. Click Overwrite / Return null and select the Overwrite as we always want to write the latest announcement using the same short name. (Return Null will return null in the output variable, which can then be tested for and action can be taken accordingly.)
     
11. Click Variable: this variable and specify the variable the Short URL will be stored in. In this example name it shortURL.
     
12. Add a Log To History List Action and specify the name of the workflow variable the Short URL has been stored in using the Workflow Lookup dialog box.

Close the workflow and create a new Announcement. When the workflow has finished, click the completed link to see the output. Click the generated URL to link to the latest announcement.

Create another Announcement, the Short URL should now link to the latest announcement.

.

Labels: Articles, MuSH, News, Products, Workflow

Posted at: 2:17 PM on 09 February 2010 by Muhimbi

In part 4 of our series of User Guide related blog postings for the Muhimbi Workflow Power Pack for SharePoint we show how to create your own methods in a WPP script in order to keep the code organised and easy to maintain.

A quick introduction In case you are not familiar with the product: The Muhimbi Workflow Power Pack for SharePoint allows custom C# or VB.NET code to be embedded in SharePoint Designer Workflows without the need to resort to complex Visual Studio based workflows, the development of bespoke Workflow Activities or long development cycles.

The following Blog postings are part of this User Guide series:

1. Language Features: Discusses the script like syntax, the generic workflow action and condition, passing parameters, returning values from a workflow and using the MyWorkflow property.
    
2. Embedding .net code in a Workflow Condition: Provides a number of examples of how to use the Evaluate Custom Code condition to carry out basic as well as complex conditional tasks.
    
3. Embedding .net code in a Workflow Action: Contains a number of examples of how to use the Execute Custom Code  to basically carry out any action you can think of in a SharePoint Designer Workflow.
    
4. Creating Custom Methods (this article): Shows how to create your own methods in your scripts in order to keep the code organised and easy to maintain.

Due to its scripting like approach, the Workflow Power Pack does not allow regular .NET methods to be created. However, by cleverly using delegates you can create your own reusable pieces of code.

To facilitate this, the following delegates can be used in addition to the normal delegates available in the .net framework. Note that this only works for C# as VB.net does not allow anonymous methods to be created.
 

delegate void WorkflowMethod(params object[] parameters);

delegate object WorkflowFunction(params object[] parameters);

delegate void WorkflowMethod<ParameterType>(params ParameterType[] parameters);

delegate ReturnType WorkflowFunction<ParameterType, ReturnType>(params ParameterType[] parameters);

 
There is no need to add these delegates to your WPP Code, they are added automatically.

The example provided below creates a generic Debug method to concatenate information to a string. This string is then returned as the workflow’s ReturnValue, from where it can be written to the Workflow History.
 

string debugString = String.Empty;

WorkflowMethod<string> Debug = delegate(string[] parameters)

{

    debugString += parameters[0] + "\r\n";

};

WorkflowFunction Calculate = delegate(object[] parameters)

{

    return (int)parameters[0] + (int)parameters[1];

};

WorkflowFunction<int, string> Calculate2 = delegate(

                               int[] parameters)

{

    return (parameters[0] + parameters[1]).ToString();

};

Debug("Hello");

Debug("World");

Debug(Calculate(1, 2).ToString());

Debug(Calculate2(3, 4));

MyWorkflow.ReturnValue = debugString;

Labels: Articles, News, Products, Workflow, WPP

Posted at: 11:36 AM on 04 February 2010 by Muhimbi

As most of our products can be used from a SharePoint workflow, it is perhaps useful to know how to tweak SharePoint’s workflow engine for high-load or other specific scenarios.

This article explains in detail what can be tuned and how it can be tuned. If you are in a rush then you can skip over the first 20%.

In summary:

• Workflow Throttle: Controls how many workflows can be processing at any one time on the entire server farm. This setting does not control how many workflows can be "In Progress" concurrently, but rather how many can be actively using the processor. When this number is exceeded, workflow instances that are started and events that wake up dehydrated workflows are queued for later processing. The default value is 15. This setting is per farm, so the number of front-end Web servers is irrelevant
  
  stsadm -o setproperty -pn workflow-eventdelivery-throttle -pv "25"
   
   
• Workflow Batch Size: Workflows, by their very nature, do not execute in a nonstop, linear fashion. Instead, they run for a little while, pause, run some more, and then pause again, continuing in this manner until the process is complete. Although an outside observer or a developer might disagree, workflows are a collection of batches and the workflow engine is simply a glorified batch controller.
  
  stsadm -o setproperty -pn workitem-eventdelivery-batchsize -pv "125"
   
   
• Workflow Timeout: The timeout setting specifies the amount of time (in minutes) in which a workflow timer job must complete before it is considered to have stopped responding and is forced to stop processing. Jobs that time out are returned to the queue to be reprocessed later. The default timeout period is five minutes
  
  stsadm -o setproperty -pn workflow-eventdelivery-timeout -pv "10"
   
   
• Workflow Timer Interval: The workflow timer interval specifies how often the workflow SPTimer job fires to process pending workflow tasks. This interval also represents the granularity of delay timers within your workflow. If a timer is set to delay for one minute, but the interval timer fires only every five minutes, the workflow delays for five minutes, not one minute
  
  stsadm -o setproperty -pn job-workflow -pv value -url http://myWssServer
   

For our products you may need to tweak Workflow Timeout for very long running PDF Conversions. Changing the Timer Interval can be useful during development when using Pause Until or Pause For workflow Activities.

.

Labels: Articles, MuSH, PDF Converter, Workflow, WPP

Posted at: 4:41 PM on 02 February 2010 by Muhimbi

The following article is an ‘open Kimono session’ where I discuss some of the internals of my company as well as our marketing program. It is my opinion that we are the victim of click-fraud, however my investigation is not 100% scientific and I have had to make some assumptions based on observations and time constraints. Please draw your own conclusion and consider everyone innocent until proven guilty. The figures, charts and tables presented in this article originate from Google’s own Analytics and AdWords software.

Update: Latest developments and responses from Google can be found at the end of this post.

After witnessing unexpected browsing behaviour from visitors who arrived on our site via a Google AdWords campaign that we ran a year ago for our PDF Converter for SharePoint, I was pretty sure that we were the victim of click-fraud.

Unfortunately, due to a lack of time and detailed figures to back up my suspicions, I decided not to pursue the matter at the time. However, after recently analysing another campaign it became clear that something suspicious is going on. Naturally it is not Google who is committing the fraud, but they are not doing enough to prevent it either.

Note that Google has settled a click-fraud related class action lawsuit in 2006 for $90M, a drop in the ocean compared to their level of revenue. The problems appear to be ongoing, read on for my findings.

What is Google AdWords?

Ever wonder how Google make tens of billions of dollars each year? One word: Advertising! AdWords is the platform that allows customers to specify keywords, bids and budget for displaying adverts next to Google’s search results as well as in-line on any website that is willing to display adverts in exchange for a share in the revenue.

When creating a campaign you can specify where the ads appear:

1. In Google’s Search Results: Based on the search terms and the keywords specified, relevant adverts are displayed next to and above the search results. The more you are willing to pay, the higher the advert will be displayed, increasing the chances of a user clicking it. Every time an advert is clicked, Google charges the advertiser a fee. As Google’s site is a trusted entity, this way of advertising is relatively fraud proof. In all fairness it appears to work exceptionally well and Muhimbi probably could not survive without it.
    
2. On the content network: This is where I suspect the majority of fraud is taking place. Anyone who can host any kind of content, e.g. a blog, can sign up as an affiliate and place Google ads in their content. Every time an advert is clicked on the content network Google charges the advertiser and part of the income is paid to the ‘owner’ of the content.
    

Although Google is putting a lot of effort in preventing fraud, the engineer in me can think of many ways to abuse the content network program, particularly using cheap labour, proxy servers and spyware like applications to simulate real user clicks.

For more detail read Wikipedia’s definition of the AdWords platform.

Muhimbi’s market and products

What makes this investigation relatively easy is the fact that we serve a niche market. All our products are aimed at corporate IT departments for use in their SharePoint environment. The campaign discussed in this article is for our Workflow Power Pack, a product that allows SharePoint Designer developers to embed C# or VB code into their workflows. A great product, but I believe there is a box shot of our product next to the definition of the word niche.

We are a small, but extremely committed company, which makes it difficult to swallow that our hard earned money appears to be used for funding criminal activity.

Normal, genuine, users

Based on our experience with other campaigns as well as ‘organic visitors’ who visit our site via external links or regular Google searches, our normal audience has the following characteristics:

1. When we send a newsletter to these users, the email rarely bounces.
    
2. They visit during weekdays. During the weekend our site has 75% less visitors compared to weekdays.
    
3. They arrive on our site via Google with relevant keyword searches or via links from external sites that are relevant to our niche.
    
4. They browse around before going to the download page. Only 41% go from the landing page directly to the download page.
    
5. Roughly half of interested visitors contact our support / sales department at some stage for further information.
    

Regular visits by day of the week. Guess which data points represent the weekend.

Evil users

The usage pattern of these alleged fraudulent users is completely different:

1. A large proportion of newsletters sent to these users bounce.
    
2. They visit every day of the week including the weekend.
    
3. They arrive on our site via questionable, unrelated sites. More about this later.
    
4. 75% of the users go directly to the download page from the landing page without getting any further information about the product.
    
5. None get in contact with our support / sales department to request any kind of information. I would like to think the information on the site is crystal clear, however this does not match the pattern we see from other products and campaigns.
    
6. They spend an average of 1 minute on the site. I am not sure if this is the minimum that Google Analytics reports, but these people are clearly ‘very committed’.
    

Visits by day of the week for pages related to this particular campaign. No weekend dips, clearly hard workers.

So why do these alleged fraudulent users go through the effort of downloading our software and registering for newsletters after clicking the advertisement, which is when they make their money? The reason behind this is that many Google campaigns as well as marketing professionals measure the success of their campaign based on conversions. For example:

1. A user enters the site via an AdWords campaign and downloads the software. This is considered a conversion and a sign of a campaign being successful. The marketing executive will get a pat on the back from the CEO and everyone is happy (initially).
    
2. A user enters the site via an AdWords campaign and subscribes to a newsletter. This is considered a conversion as well resulting in CEO –> pats marketing on back–> Happy –> time expires –> Sad –> Fired –> Divorce –> Death (See the pain these people are causing!)
    

Clever marketing people and, if configured that way Google AdWords, measure conversions and allocate more budget to sites that generate these conversions. A good reason for fraudsters to simulate some activity after clicking an advertisement.

Which sites are the worst offenders?

It should come as no surprise that sites that allow anonymous users to host their own content and insert Google advertisements are the worst offenders as it is almost impossible to trace these people. From a geographic perspective it appears that Chinese web sites are the worst, but many other countries are just as bad.

Our advert has been displayed 2.5 million times on 767 sites over the course of one month. 611 different sites have referred at least one visitor. Out of those sites I consider about 60 sites relevant in the loosest sense of the word (Intentionally or not, eggheadcafe.com for example is legit although until recently very mischievous in the way they presented and positioned their advertisements to make them look like clickable answers to questions. They still do it on some threads, but not as bad as it used to be).

76 sites had an amazing 100% click through ratio, 213 with a CTR of more than 20%, 345 with more than 10% and 450 with more than 1% (which is still an amazing rate considering mail.google.com has a 0.02% CTR).

Sites such as divxphoto.com (domain for sale) I consider to be irrelevant as the domain is for sale and doesn’t actually display any advertisements(!!!!!) Most of the other sites on the list can be categorised as domain for sale, dodgy software download site, driver download site or rubbish content aggregation site.

Listed below are the top 15 sites by highest number of advertising clicks.

Green: Most likely legit - Amber: Likely to be illegitimate - Red: Almost certainly illegitimate

Listed below are the top 15 Sites by Click Through Ratio with more than 25 impressions (otherwise the table would contain 76 sites with a 100% CTR after a single impression, which is rather useless).

Green: Most likely legit - Amber: Likely to be illegitimate - Red: Almost certainly illegitimate

 
I realise that detecting and solving click fraud is much more difficult than actually causing it, especially without access to key information such as site demographics, visitor behaviour, click streams and conversion data. On the other hand, as Google Analytics tracks the Muhimbi Site, they actually have most of this data. I will present my findings to Google and give them a chance to respond and hopefully improve the situation for everyone. Perhaps some kind of validation system or list of ‘trusted sites’ could be created by Google.

As a Google shareholder I wonder how much of Google’s income actually comes from this kind of alleged criminal activity. According to Google: “…we manage the problem of invalid clicks very well. We have a large team of expert engineers and analysts devoted to it. By far, most invalid clicks are caught by our automatic filters and discarded *before* they reach an advertiser’s bill. And for the clicks that are not caught in advance, advertisers can notify Google and ask for reimbursement.”

This situation cannot continue any longer. I am naturally upset that my company appears to be the victim of fraud, but what about the thousands (millions?) of other advertisers who do not have the knowledge or resources to detect fraud? It should not be up to the customers to research and report fraud, Google should step up its game and clean up its act, no matter how difficult or painful it is. 

So, is Google guilty of fraud? I seriously doubt it, however they appear to be profiteering from other people’s criminal activity in a manner not dissimilar to the way illegal media sharing sites are behaving. “We are not doing anything illegal, we can’t help it that other people upload illegal movies / music / software / <insert excuse here> to our site even though it has clearly been designed for this purpose.”

… Not good enough. To be continued.

02-Feb-2010 - Update 1: We are clearly not the only party experiencing click-fraud. For more information visit the links below:

• Internet HoneyGrid reveals 95% of User Generated Content is spam or malicious.
• Can A Company Selling PPC Ads Eliminate Liability For Click Fraud By Just Saying So?
• Industry Click Fraud Rate at 15.3 Percent for Q4 2009
• ClickForensics – Click Fraud Index
• Forbes - Google Faces The Slickest Click Fraud Yet
• How To Avoid Google Adsense Click Fraud In Any Online Business
• Advertising Click Fraud Rampant Online?
• Google Ignoring (Benefiting From) Click Fraud?
• Google Click Fraud Inflates Conversion Rates and Tricks Advertisers into Overpaying

Report Google click-fraud here.

18-Feb-2010 - Update 2: Google have responded and claim that only a small percentage of the clicks are fraudulent. The remaining clicks are all part of normal user activity. It appears their response is largely automated so I have replied back asking for further details as I don’t accept their findings. I find it astonishing how much they downplay the issue of click-fraud. Apparently it is up to me to manually exclude domains that I consider not to be relevant. This is just laughable.

19-Feb-2010 – Update 3:

• Typos may earn Google $500m a year – NewScientist, ZDNet, Harvard University Research
• Google Sued in Domainer Lawsuit--Vulcan Golf v. Google

.

Labels: Articles, News

Posted at: 12:18 PM on 26 January 2010 by Muhimbi

We get a great amount of interest from our existing PDF Converter customers for the Workflow Power Pack for SharePoint. I am not really surprised as there is a lot of of synergy between the two.

Last week I described how to configure PDF Security settings from a SharePoint workflow. This time I’ll describe another common request; adding a watermark to a PDF File.

A quick introduction for those not familiar with the product: The Muhimbi Workflow Power Pack for SharePoint allows custom C# or VB.NET code to be embedded in SharePoint Designer Workflows without the need to resort to complex Visual Studio based workflows, the development of bespoke Workflow Activities or long development cycles.

The solution presented below executes a workflow whenever a PDF file is added or updated. As part of the workflow it loads portrait and landscape based watermarks and applies the relevant template to each page of the document. The workflow author can specify if the watermark goes in the foreground or the background, the SharePoint location of the watermarks as well as the level of transparency. As the code is well documented it is easy to make further changes and customisations, e.g. the PDF file that needs to be watermarked may not be the file the workflow is acting on.

 
Create the workflow as follows:

1. Download and install the Muhimbi Workflow Power Pack for SharePoint.
    
2. Download and install the Muhimbi PDF Converter for SharePoint.
   Note that you need version 3.1.2.18 or newer, older versions do not allow watermarking.
    
3. Download this article’s source code and sample watermarks.
    
4. We need to be able to access functionality in the Muhimbi.SharePoint.DocumentConverter.PDF and System.Drawing assemblies. Add these references to the relevant Web Application using the Workflow Power Pack Central Administration screens as described in the Administration Guide. Make sure to place each reference on a new line.
     
5. Make sure you have the appropriate privileges to create workflows on a site collection.
    
6. Create a new workflow using SharePoint Designer.
    
7. On the Workflow definition screen associate the workflow with the Shared Documents library, tick the boxes next to both ‘Automatically start….’ options and proceed to the next screen.
    
8. We only want to act on files of type PDF. Although we could have put this validation in the code, in this example we use a workflow condition for it so add a Compare Any Data Source condition and:
    
   a.  Click on the first value followed by the display data binding (fx) button.
   b.  Select Current Item as the Source and select File Type in the Field. Click the OK button to continue.
   d.  Click on the second value and enter pdf.
    
9. Click the Actions button and insert the Execute Custom Code action.
    
10. Optionally click parameter 1 and enter a relative or absolute destination path. Leave the parameter empty to save the watermarked file on top of the existing PDF file. For details about how paths are handled, see this post and search for the words ‘this url’.
     
11. Insert the C# based code embedded in step #3’s download (also listed below) by clicking this code.
     

    /*********************************************************************************************

                                 Muhimbi PDF Converter - Watermarking

                  Copyright 2010, Muhimbi Ltd - www.muhimbi.com - All rights reserved

     The following code shows a simple way of adding a watermark to existing PDF Files. 

     Error and permission checking as well as other minor features have been omitted for the sake 

     of brevity and clarity.

     Ideally PDF Conversion, applying security and watermarking is executed in the same step, see 

     http://www.muhimbi.com/blog/2010/01/configure-pdf-security-from-sharepoint.html

     This code requires Muhimbi’s PDF Converter and Workflow Power Pack to be installed.

    *********************************************************************************************/

    using System.Drawing;

    using System.IO;

    using Syncfusion.Pdf;

    using Syncfusion.Pdf.Parsing;

    using Syncfusion.Pdf.Graphics;

    using Muhimbi.SharePoint.DocumentConverter.PDF;

    SPFile spSourceDocument = MyWorkflow.Item.File;

    string destinationFileName = spSourceDocument.Name;

    string destinationFolderName = MyWorkflow.Parameter1 as string;

    string watermarkDocumentPortraitPath = @"Shared Documents/Watermarks/WatermarkA4Portrait.pdf";

    string watermarkDocumentLandscapePath = @"Shared Documents/Watermarks/WatermarkA4Landscape.pdf";

    // ** z-order and transparency of the watermark

    bool watermarkInBackground = true;

    float watermarkTransparancy = 0.25f;

    // ** Load the document and watermarks

    PdfLoadedDocument sourceDocument = new PdfLoadedDocument(spSourceDocument.OpenBinary());

    SPFile spWatermarkDocumentPortrait = MyWorkflow.Web.GetFile(watermarkDocumentPortraitPath);

    PdfLoadedDocument watermarkDocumentPortrait = new PdfLoadedDocument(

                                                    spWatermarkDocumentPortrait.OpenBinary());

    SPFile spWatermarkDocumentLandscape = MyWorkflow.Web.GetFile(watermarkDocumentLandscapePath);

    PdfLoadedDocument watermarkDocumentLandscape = new PdfLoadedDocument(

                                                    spWatermarkDocumentLandscape.OpenBinary());

    // ** Create the PDF Templates

    PdfTemplate watermarkTemplatePortrait = watermarkDocumentPortrait.Pages[0].CreateTemplate();

    PdfTemplate watermarkTemplateLandscape = watermarkDocumentLandscape.Pages[0].CreateTemplate();

    // ** Iterate over all pages and apply watermark

    foreach (PdfPageBase page in sourceDocument.Pages)

    {

        // ** Is the current page portrait or landscape?

        PdfTemplate watermarkTemplate = null;

        if (page.Size.Width > page.Size.Height)

            watermarkTemplate = watermarkTemplateLandscape;

        else

            watermarkTemplate = watermarkTemplatePortrait;

        // ** Place watermark behind or in front of text?

        if (watermarkInBackground == true)

        {

            PdfTemplate pageTemplate = page.CreateTemplate();

            page.Layers.Clear();

            PdfGraphics g = page.Graphics;

            g.SetTransparency(watermarkTransparancy);

            g.DrawPdfTemplate(watermarkTemplate, PointF.Empty, page.Size);

            g.SetTransparency(1f);

            g.DrawPdfTemplate(pageTemplate, PointF.Empty, page.Size);

        }

        else

        {

            PdfGraphics g = page.Graphics;

            g.SetTransparency(watermarkTransparancy);

            g.DrawPdfTemplate(watermarkTemplate, PointF.Empty, page.Size);

        }

    }

    // ** Construct the path and file to write the watermarked PDF file to.

    if (string.IsNullOrEmpty(destinationFolderName) == true)

        destinationFolderName = spSourceDocument.ParentFolder.Url;

    SPFolder destinationFolder = Utility.GetSPFolder(destinationFolderName, MyWorkflow.Web);

    string destinationFilePath = string.Format("{0}/{1}", destinationFolder.Url,

                                               destinationFileName);

    SPWeb destinationWeb = destinationFolder.ParentWeb;

    SPFile spDestinationFile = destinationWeb.GetFile(destinationFilePath);

    // ** If a document library requires manual checkout and the file is not checked out, then 

    // ** check the file out before uploading.

    if (spDestinationFile.Exists && spDestinationFile.Item.ParentList.ForceCheckout &&

        spDestinationFile.CheckOutStatus == SPFile.SPCheckOutStatus.None)

    {

        spDestinationFile.CheckOut();

    }

    // ** Add the file to the site including the meta data

    using (MemoryStream watermarkedFile = new MemoryStream())

    {

        sourceDocument.Save(watermarkedFile);

        spDestinationFile = destinationWeb.Files.Add(destinationFilePath, watermarkedFile,

                                                     spSourceDocument.Item.Properties, true);

    }

    // ** Check the file back in if this script was responsible for checking it out.

    if (spDestinationFile.Item.ParentList.ForceCheckout == true)

    {

        spDestinationFile.CheckIn("Auto check-in after PDF watermarking.");

    }

     
12. Click the Actions button, select Log to History List, click this message and enter File watermarked.
     
13. Close the Workflow Designer.
     
14. In the Shared Documents library create a sub folder named Watermarks and copy the 2 watermark files that are part of step #3’s download into it. If the watermarks are named differently or stored in a different location then update the paths in the script accordingly. Note that the sample watermarks are exactly A4 sized but they work fine on differently sized documents.
     
15. Update an existing PDF or add a new PDF file to your library to trigger the workflow and apply the watermarks.
     

Naturally this is just a simple example. Feel free to play around with the code, change which parameters are passed into the workflow, modify where watermarked files are written to or add dynamic text as a watermark. Leave a comment below if you are trying to do anything specific.

.

Labels: Articles, PDF Converter, Products, Workflow, WPP