Back!

so it has been a very very long while since i put anything on this blog, but well things has been quite crazy :)


anyway, lately i have been doing alot of investment in WCF/WSE. things with regards to specs and definitly deep dive in WCF.

i have been always one of those guys who had lots of problems with [WebMethod] framework. simply the fact that you don't create your interface sepratly as in you define it somewhere then consume or implement in fact you design & implement your prog interface by tagging methods on a class with [WebMethod] !

additionally (just a little example) you have no control over WSDL which is quite a shame cause there is alot of intersting stuff to be done on this front

anyway the 2 combined factors created a mess of web services. with badly designed WSDL & web services in general





got to admin: WSE team is the first from microsoft side to acknowlege the defect and fixed it in thier own implementation led by smart people like Yasser Shohod. and it is simple you create interface following

[WebServiceBinding( Name = "StockService", Namespace = "http://stockservice.contoso.com/wse/samples", ConformsTo = WsiProfiles.BasicProfile1_1, EmitConformanceClaims = true)]
interface IStockService
{
//Operation goes here
}

then you create your service implementation


[WebService( Namespace = "http://stockservice.contoso.com/wse/samples")] public class StockService : IStockService
{
//SERVICE HERE
}


that was primarly aiming towards implementing services outside IIS scope.


later on WCF took this further by using servicecontract terminilogy (very smart choice) and definitly instance managment.



more on this on later posts

Nasty Serializer Bugs

Ok one day, you design your schema working those multi dimention arrays thinking if it works in your orchs it shouldln't be a problem anywhere else.

the hard aquired experience is no, you gonna fall into a trap called serializer bugs :(

.NET serialization though the most powerful tool currently existing it has some heckups, one of them is the multi-dimention. on win 2000 box you will get "File/assembly not found and some random name" on win2003 you will get something like "can not generate XXX class", some wacky error telling you nothing.

the moral of the story:
- Keep your schemas simple
- Test them with Chris Sell's Serlization Compiler (www.SellsBrothers.com)
- Read http://msdn.microsoft.com/library/en-us/dnxmlnet/html/trblshtxsd.asp, and learn of it

happy 'schema design' ;)

Kal

Configuration Data!

it has been kind slow the past couple of weeks, maybe because it is summer :)

anyway today i am talking about configuration: first let us start with who consumes configuration

- Adapters
- Pipelines
- Orchs

let us start one by one
Your adapters will need to consume some configuration data, starting from debug flags to some processing related flags. the first question is who will run your adapters is it BTS services or IIS or your own custom process:

• if it's BTS: You are left either with editing BTSNTSVC.EXE.CONFIG to have your configuration, but carefull not to mess BTS configuration. or you can use Enterprise Library Configuration Block which work just fine.
• if it's IIS: Something like the SOAP or HTTP adapter, well alot of us neglects the fact that ASP.NET runtime will run our code (adapter in this case) which means youn can leaverge ASP.NET runtime configuration tools (Web.Config) or even use the HTTPContext classes.
• if it is your process: well you can do whatever you want but a strong advice go for Configuration block it really comes handy, and hey it is free ;)

as for pipelines: well the same as the adapter really applies on pipelines. which is good because the same approch can be resued all over again.

finally the orchs: if it is simple configuration (Text fields) i stronglly recomment using Const members on exeternal classes since it is very fast however, beaware that .NET compilers replaces those with the actual text upon compiling - that is a tip from a good guy i know :) - so if it is better using static properties as the compilers doesn't replace those. again if you have to save configuration you can edit BTSNTSVC.EXE.CONFIG or use the configuration block.

well that was just some thoughts that i did like to share with you.. hope they help.

Kal

Tips for performance

1- Use promoted props only for routing and subscription evaluation, these ones are slow, if you want to use property inside an orch use distingueshed fields they are faster.

2- Pipelines: don't use XMLDocument object to load the document instead use firehose style cusrors to parse the XML, keep these pipelines as fast as possible as they are the real perforamance bottlenecks

3- Watch out for serialization: BTS serializes your process on various conditions, however you can control that read my "Fast BPI, Faster Than What?" blog entry for options.

4- infact try not to use orchs :), yes if you can replace that with content based routing then do that. the idea is to pay the lowest perfomance cost possible.

5- Watch out for parrlle shapes, they don't look as fast as you think. careful use them as possible.

6- use external code carefuly, BTS in itself is a very fast engine (nice done). but the code you write my not be designed to be as fast so only do that when you need to

7- and if you need to use statless objects for example static methods on classes.

8- Maps: XLTS functiods are faster than external assmblies, need to say why?

7- minimize the state you save: ok everything in an orch is a state, including variables messages correlation sets (ofcource scoped either by orch or by scope shapes) everything gets saved into SQL using serialization, so minimizing this will make BTS consume less memory and ofcource the call to SQL will be smaller.

8- Use transactional carefully, compensating transaction are by nature slow

9- use correlationsets carefully, currently i couldn't see anything slower than that in BTS 2004, the reason is calls to SQL either to put the subscription record or to evaluate it :(

10- oh yes this one is VERY nice, use XMLRecieve pipleine in recieve ports, use passthru in sending, reason: unless you are depromiting properties back to the message then use passthru they are much faster, use XMLRecieve in recive because they promote the propties.

Did i start that Orch before?

Okay,


So You want to know if you started this particular instance of an orch before or not; here’s the situation: you have a business process, upon reaching a certain condition you spawn off an orch, simple eh? Remind me of the old threading days however here is the challenge your business process is layed over in away that this condition might repeat itself, raises a good question.. have we started this orch before or not. In a server programming model oyou would have kept a handle for the thread or (app domain or process) somewhere in a static variable and always check it before starting it.


However in BTS it is completely stateless, I hear the voice in my head (as in yours) saying dude save the state in the database some where, well looks like the perfect solution (the cost of one straight forward DB call isn’t really much after all). Come to think of it we do have a state DB by default (yes I mean this MsgBox DB), can we just use it and check against it. The answer is yes using

- Direct DB access (DON’T DO THAT AT HOME style, again).
- Using subscription engine! And that is where I am going to leverage on

Anyone familiar with BTS subscription viewer, that small utility that we use to debug our business processes shows us what orchs ports is waiting for what(and their correlations as well). Reflecting that tool showed that it is using Biztalk.RulesEngine assembly.. NIIICE!. We just need to get the same data dump filter it and check against it.

Let us walkthrough the subscription viewer: it is an EXE application.
Single window has a
- Data-grid (displays the subscription record)
- A tree view that shows the details of the subscription (which port, which web method name which correlation and it is value etc…).

Filtering the code showed a single method with the name of LoadSubscriptions that loads all the subscription the code is like:

public void LoadSubscriptions()
{
this.m_Subscriptions.Clear(); // Clears the datagrd
this.PredicateLists.Nodes.Clear(); // clears the tree view
SubscriptionRuleStore store1 = new SubscriptionRuleStore("Subscription Store"); // connect to the store
this.m_rs = store1.GetRuleSet(new RuleSetInfo(this.m_GroupName, 0, 0)); // GroupName is always -> = "BizTalk Group"; (in the CTOR of the form)
foreach (Rule rule1 in this.m_rs.Rules.Values) // enumerate the list – previous call is too slow!
{
// Basiclly what is happening here is creating a data table contains all the subscription and binding the grid to it.
string text1;
DataRow row1 = this.m_Subscriptions.Tables["Subscriptions"].NewRow();
row1["SubscriptionID"] = rule1.Name;
row1["Priority"] = rule1.Priority;
UserFunction function1 = (UserFunction) rule1.Actions[0];
ClassMemberBinding binding1 = (ClassMemberBinding) function1.Binding;
Constant constant1 = (Constant) binding1.Arguments[0];
row1["Name"] = constant1.Value.ToString();
constant1 = (Constant) binding1.Arguments[1];
if ((text1 = constant1.Value.ToString()) == null)
{
goto Label_0186;
}
text1 = string.IsInterned(text1);
if (text1 != "3d7a3f58-4bfb-4593-b99e-c2a5dc35a3b2") // Some stuff are canceled from the view, my guess internal BTS pluming stuff
{
if (text1 == "59f295b0-3123-416e-966b-a2c6d65ff8e6")
{
goto Label_0162;
}
if (text1 == "226fc6b9-0416-47a4-a8e8-4721f1db1a1b")
{
goto Label_0174;
}
goto Label_0186;
}
row1["ServiceType"] = "MSMQT";
goto Label_019D;
Label_0162:
row1["ServiceType"] = "EPM";
goto Label_019D;
Label_0174:
row1["ServiceType"] = "XLANG";
goto Label_019D;
Label_0186:
row1["ServiceType"] = constant1.Value.ToString();
Label_019D:
constant1 = (Constant) binding1.Arguments[2];
row1["ServiceID"] = constant1.Value.ToString();
constant1 = (Constant) binding1.Arguments[3];
row1["InstanceID"] = constant1.Value.ToString();
constant1 = (Constant) binding1.Arguments[4];
row1["PortID"] = constant1.Value.ToString();
constant1 = (Constant) binding1.Arguments[5];
if ((text1 = constant1.Value.ToString()) != null)
{
text1 = string.IsInterned(text1);
if (text1 != "0")
{
if (text1 == "1")
{
goto Label_029A;
}
if (text1 == "2")
{
goto Label_02AC;
}
if (text1 == "3")
{
goto Label_02BE;
}
}
else
{
row1["Enabled"] = "Disabled";
}
}
goto Label_02CE;
Label_029A:
row1["Enabled"] = "Enabled";
goto Label_02CE;
Label_02AC:
row1["Enabled"] = "Deactivated";
goto Label_02CE;
Label_02BE:
row1["Enabled"] = "AdminDeactivated";
Label_02CE:
constant1 = (Constant) binding1.Arguments[6];
row1["ApplicationName"] = constant1.Value.ToString();
this.m_Subscriptions.Tables["Subscriptions"].Rows.Add(row1);
}
if (this.m_rs.Rules.Count > 0)
{
this.SubDataGrid.Select(0);
this.SubDataGrid.CurrentRowIndex = 0;
this.SubDataGrid_CurrentCellChanged(null, null);
}
}





Okay the previous code load s the subscriptions but does nothing else, upon cell change on the grid it displays the detils of the subscription as

this.PredicateLists.Nodes.Clear();
int num1 = 0;
int num2 = 0;
Rule rule1 = (Rule) this.m_rs.Rules[this.SubDataGrid[this.SubDataGrid.CurrentRowIndex, 1]];
if (rule1.Conditions is LogicalAnd)
{
this.PredicateLists.Nodes.Add(new TreeNode("AndGroup0"));
LogicalAnd and1 = (LogicalAnd) rule1.Conditions;
for (num1 = 0; num1 < and1.Arguments.Count; num1++)
{
this.PredicateLists.Nodes[0].Nodes.Add(this.GetPredicateDisplayString(and1.Arguments[num1]));
}
}
else if (rule1.Conditions is LogicalOr)
{
LogicalOr or1 = (LogicalOr) rule1.Conditions;
for (num1 = 0; num1 < or1.Arguments.Count; num1++)
{
this.PredicateLists.Nodes.Add(new TreeNode("AndGroup" + num1));
if (or1.Arguments[num1] is LogicalAnd)
{
LogicalAnd and2 = (LogicalAnd) or1.Arguments[num1];
for (num2 = 0; num2 < and2.Arguments.Count; num2++)
{
this.PredicateLists.Nodes[num1].Nodes.Add(this.GetPredicateDisplayString(and2.Arguments[num2]));
}
}
else
{
this.PredicateLists.Nodes[num1].Nodes.Add(this.GetPredicateDisplayString(or1.Arguments[num1]));
}
}
}
else
{
this.PredicateLists.Nodes.Add(new TreeNode("AndGroup0"));
this.PredicateLists.Nodes[0].Nodes.Add(this.GetPredicateDisplayString(rule1.Conditions));
}
this.PredicateLists.ExpandAll();
}


Cool eh? Well what you need to know is your subscription will always contain data, additions to what you have in mind, stuff that has to do with Port IDs. However if you know the correlation attribute name, value, orchestration name and assembly name it is enough information that you can use to answer the question raised (did I start this one before or not). Be aware you may have multiple subscription for the same correlation if it is being used so many times in your process on different ports so make sure that you filter using the correct information.

Final words:
- The subscription viewer in itself is a slow tool because it dumps all the subscription. I spent sometime trying to find any way to filter the data rather than killing the database but couldn’t find any so far. I am sure there is since BTS is very fast evaluating messages against subscriptions.
- SubscriptionRuleStore is not a supported by BizTalk so whatever you write here is write it at your own risk.

Toying with those maps

So what i am trying to figure out or prove is XSLT maps are faster than standard BTS maps outputs, reason i believe so is XSLT maps will use standard .NET transformation classes which based on firehose cursor style to process XML messages

however standard maps are basiclly XML document that BTS has a way to process, what i did is i reflected the code for a map that uses XSL and here is what i found
[SchemaReference("LinkDev.BizTalk.Adapters.OracleStaticAdapter.Schemas.CommandExecuteRequest", typeof(CommandExecuteRequest)), SchemaReference("MinistryOfCommerce.eInvestor.BizTalk.Schemas.SuperApprovalSchema", typeof(SuperApprovalSchema))]
public sealed class mapAppFeesSUperAppIn2ExecuteReq : TransformBase
{
// Methods
public mapAppFeesSUperAppIn2ExecuteReq();

// Properties
public override string[] SourceSchemas { get; }
public override string[] TargetSchemas { get; }
public override string XmlContent { get; }
public override string XsltArgumentListContent { get; }

// Fields
private const SuperApprovalSchema _srcSchemaTypeReference0 = null;
private const string _strArgList = "\n";
private const string _strMap = " \n \n \n \n\t --- Rest of XSLT file that i used for the map";
private const string _strSrcSchemasList0 = "MinistryOfCommerce.eInvestor.BizTalk.Schemas.SuperApprovalSchema";
private const string _strTrgSchemasList0 = "LinkDev.BizTalk.Adapters.OracleStaticAdapter.Schemas.CommandExecuteRequest";
private const CommandExecuteRequest _trgSchemaTypeReference0 = null;
}




while trying to reflect a standard map i found:
[SchemaReference("MinistryOfCommerce.eInvestor.BizTalk.Schemas.eInvestorResponseMessage", typeof(eInvestorResponseMessage)), SchemaReference("MinistryOfCommerce.eInvestor.BizTalk.Schemas.eInvestorTransactionMessage", typeof(eInvestorTransactionMessage))]
public sealed class mapInternal2Response : TransformBase
{
// Methods
public mapInternal2Response();

// Properties
public override string[] SourceSchemas { get; }
public override string[] TargetSchemas { get; }
public override string XmlContent { get; }
public override string XsltArgumentListContent { get; }

// Fields
private const eInvestorTransactionMessage _srcSchemaTypeReference0 = null;
private const string _strArgList = "\n";
private const string _strMap = " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n";
private const string _strSrcSchemasList0 = "MinistryOfCommerce.eInvestor.BizTalk.Schemas.eInvestorTransactionMessage";
private const string _strTrgSchemasList0 = "MinistryOfCommerce.eInvestor.BizTalk.Schemas.eInvestorResponseMessage";
private const eInvestorResponseMessage _trgSchemaTypeReference0 = null;
}



so conclusion so far are:
- even standard BTS maps generates XSLT and being pocessed slightly different check out those var:var statments.
- I still belive that XSLTs are faster and they gave u more flixbility more on this now.
- If i want to create a simple map i will use standard BTS map wizard, while if i want a complex (will be used alot) i will go for XSLT


Kal

Fast BPI, Faster Than What?

Regular weekend cappuccino J trying to get thoughts straight for that blog entry, okay what I am talking about. I am talking about BTS 2004 Orchs performance and optimization.

I happened to be in one of these projects where we are delivering an integrated “fully configurable” workflow, you know the drill external data store contains all the configuration data this configuration data affects your orchs in a couple of areas
- Message process – should your workflow process a just received message or kick it back returning some business validation rule violation error – yeah those errors that you keep in Strings.cs or external resources files

- Decisions shapes – should your orch do X or Y


Anyway during the implementation I found out about a couple of interesting things that would

- Make your life easier during the development
- Let you answer questions from other techy’s/whoever telling you hey that is slow I can write C# code that would do better job and more importantly faster.



Development:
As you have noticed developing orchs using VS.NET 2003 is nasty especially – That one is much, much better in BTS 2006 code named PathFinder as I have learned - if you are implementing a lot of ports and messages in/out. The reason behind that is GDI L. Somewhere in the designer it consumes a lot of memory I found out that:
- Develop using Windows XP or 2003 it – the designer - is more optimized on them (don’t ask me why I am trying to find out why still)
- If you have to do it on win 2000 then get that SP1 it really helps
- Don’t connect to local SQL, Deployment and enlisting generates a lot of traffic on local SQL so why not using one of these big/ fat development servers instead of connecting locally
- Always, always close port surface like:



Reason behind that is it really puts GDI @ ease to eliminate these lines connecting your shapes and ports

- If you are using Virtual PC to do the good stuff, then put it – VMD files - on a different HDD if you are running on a tower pc or put it on USB 2 external HDD if you are running on a laptop – yes these works like a charm

Runtime:
Okay scaling up or out your servers is pretty straight forward, adding more servers and how to add them is published everywhere on MSDN and Microsoft web site. But what I am trying to introduce is ability to get these orchs to act faster and do more.


Let me tell you this the problem that might make your orchs slower isn’t executing code within the orchs it is the state maintenance, by that I mean the fact that BTS saves the orchs during execution so if things went wrong it can compensate, execute from last successful step or whatever you configure it.

Most of the code generated out of orchs is maps call or send/receive calls the heavy calls – other than state maintenance is during sending or receiving (hence pipeline/adapters) can really damage your performance expectation but I am saving that talk for later blog entry.

Before I go on I want to introduce to concepts related to orchs processing within BTS
- Serialization: is done when BTS put a mark on your orch instance, think of it as the last successful action made during you’re your processing, this happens a lot in Send/Receive loops etc..
o How it happens: BTS orch code genereated is just a .NET class that implements ISerializable interface or other seriliazation mechanism, during the execution BTS decides it needs to mark it, it serializes everything to the MSGBOX DB. And it doesn’t take anything out of BTS Host’s memory. Pretty smart stuff eh? J loved that one btw
- Dehydration: is done when BTS decides ok this particular instance is not needed in the memory any more (happens in listen, delay, receive shapes – other than activation). Later on when BTS decides to awaken – hydrates – this instance based on message received, delay is over – those one are message also called timer messages- , more on that later on other blogs.
o Ho it happens: it happens using serialization as well but this time BTS takes everything out of the host’s memory.


As you have noticed both operations entails heavy calls to SQL stored procs, unfortunately you can not control dehydration process as it is totally controlled by BTS – which is good to tell you the truth – however the serialization process is controllable using few tricks.

Btw I found this post to control the dehydration but it is don’t do it at home style



Back to our talk, BTS never serializes scoped operations reason is message processing so scope shapes can be your best friend if you used them well:

Consider the following picture: it is send receive shapes in a scope (BTS will not go to DB in scopes, I have tried atomics and none transactional none synchronized scopes and it works like a charm). This way you minimize calls to SQL hence faster processing.



NOTE: doing that will not let BTS mark the execution so if it is failed somewhere in the middle of the scope it will start the execution before the scope. Hence do that carefully on top of my mind I can tell you
- do that when there is no harm repeating the calls (like Update statements)
- do that in loops if there is no harm repeating your loops starting the counter @ 0.
- Do that if your adapters are transactional.
- Don’t do that when your sending messages to external entities (you never trust how they process repeated messages)
- Don’t do that during calls to web services (unless they implement some transactional mechanism).


Other tips related to performance is parallel execution
- No I am not saying use it, instead don’t use it in message processing. Use them if you are doing some execution that has nothing to do with sending or receiving message. Reason is using that will most probably require some synchronization shapes which hurts your performance. (in addition imagine the number of calls to SQL if you have a couple of branches that send/receive messages)

If you want to test how many times BTS serializes your orch instances create a serializable class have it as amember variable in your orch and let it save counter for each time your serialization ctor called, then try with different shapes (loops receive delays and so on) promise, astonishment!

I will keep this post short will stop here, later I will talk about messages/schemas I want to tell you about optimizing your schemas for performance (that will entails talking about pipelines)


Feel free to communicate with me on khnidk@gmail.com

Enjoy BPI,
Kal