Re: [WebDNA] Searching for multiple bits of data in one field or use multiple fields for each bit of data?

This WebDNA talk-list message is from

2016


It keeps the original formatting.
numero = 113032
interpreted = N
texte = 620 --Apple-Mail=_84F953D3-9AAF-4924-826B-6B34A4888859 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Some historical background, for those who=E2=80=99ve been with the = product since the WebCatalog days: We used a bitwise ranking system similar to this for keyword-based = search results. WebCatalog=E2=80=99s =E2=80=98native=E2=80=99 ranking = (meaning the order in which it returned search results if you *didn=E2=80=99= t* specify a field to sort on) would take all of your search keywords = and assign them a bit, going from left to right across a 32-bit unsigned = integer. Each keyword that matched would get a 1, and ones that didn=E2=80= =99t match got a 0. The heuristic assumption being that people would = tend to type the most-important keywords first, so those became the = higher-order bits in the integer. Each record in the table got a 32-bit = score this way, and when we were finished scanning the entire table, we = merely sorted the results by that 32-bit unsigned integer. Voila: = best-matching records were always at the top. Not exactly Google pagerank, but as heuristics go, it got the job done = fairly well ;) > On Sep 30, 2016, at 10:47 AM, Patrick McCormick = > wrote: >=20 > It=E2=80=99s actually called a bitwise operation - for geek history = buffs. Let=E2=80=99s say you have 5 different conditions you=E2=80=99re = monitoring (I=E2=80=99m just picking a random number of random tests): >=20 > 1) temp above limit > 2) ping fail > 3) http fail > 4) dns fail > 5) disk space alarm >=20 > First, all of these must be yes/no, binary conditions. For temp = limits, the set point and control loop are elsewhere, this just looks at = the resulting alarm as true/false. >=20 > If I put those conditions into a single field called conditions, and = none are in alarm, the result would simply be five zeros. If any one = condition alarms, the position of that number would indicate what = condition failed. if #1, 2, & 3 were all in alarm, the result would = 11100, or 7, in base 10. >=20 > You can either create a db to store all 32 possible different = combinations of alarms: >=20 > valuetemppinghttpdnsspace > 0fffff > 1tffff > 2ftfff > etc. >=20 > You can write a function to trigger whenever the [conditiona] variable = is greater than zero, then lookup the base 10 value of [conditions] in = that db to see what the alarms are. >=20 > But because the data is stored in binary, you can use a much quicker = process to figure out which conditions are in alarm. Take a look at the = [listchars] command. You can do = this: >=20 > [listchars chars=3D[conditions]] > [index]: [char] > [/listchars] >=20 > That produces: >=20 > 1: 1 > 2: 1 > 3: 1 > 4: 0 > 5: 0 >=20 > And, if you just store a simple list of numbered alarms, the lookup = happens in a fraction of the time because your alarm db only has as many = entries as alarms, rather than all possible combinations. >=20 >=20 >=20 > On Sep 30, 2016, at 12:23 PM, dale > wrote: >>=20 >> Hello Patrick, >>=20 >> Interesting idea. I've not tried that in the past. Any possible = examples of how you would do that? It definitely sounds like a good way = to go. >>=20 >> Thank you, >>=20 >> Dale >>=20 >> On Fri, Sep 30, 2016 at 10:06 AM, Patrick McCormick = > wrote: >> I think you should store each alarm condition as one bit in a binary = number long enough to accommodate all alarm conditions. Then, each base = 10 value of that binary number represents a unique alarm or combination = of alarms. >>=20 >>=20 >> > On Sep 29, 2016, at 6:42 PM, dale > wrote: >> > >> > I searched the talk archives, though I probably didn't use the = proper search terms. >> > >> > The system for work I will be building is kind of like a fleet = management system. It isn't tracking the vehicles, but a computer and = camera system we install and maintain on the vehicles. >> > >> > One of the things I wanted to do is to put the various problem = codes (e.g. C HD PCB SSD) in a single field separated by a comma. My = thought is that it would be easier to search for systems that had a = specific problem and/or combination of problems. >> > >> > As it has been a few years since coding in WebDNA, I'm not sure if = this is the best way or not. We do come up with new codes as time passes = based on new systems. >> > >> > Would you recommend using a single field for various codes as = listed above where I would only have to change the searches or using a = hdr file to make adding additional codes a bit easier? >> > >> > I really haven't searched for multiple bits of data in a single = field before. >> > >> > Ways in which this would be used: >> > Someone checking the systems remotely would enter the code for that = system >> > Techs fixing the system would need to remove the appropriate codes = for what they fixed >> > Running reports to determine which systems have certain issues >> > >> > Additionally, each time the techs create a service ticket to make a = repair, I will save a copy of the the status of the system along with = their notes in a history database so we will have a life-cycle of the = system. Those records would only be editable by certain admin accounts. >> > >> > Thank you in advance for any recommendations as to the best way to = handle this. >> > >> > Dale >> > >> > --------------------------------------------------------- This = message is sent to you because you are subscribed to the mailing list . = To unsubscribe, E-mail to: archives: = http://mail.webdna.us/list/talk@webdna.us = Bug Reporting: = support@webdna.us >>=20 >> --------------------------------------------------------- >> This message is sent to you because you are subscribed to >> the mailing list >. >> To unsubscribe, E-mail to: > >> archives: http://mail.webdna.us/list/talk@webdna.us = >> Bug Reporting: support@webdna.us >>=20 >> --------------------------------------------------------- This = message is sent to you because you are subscribed to the mailing list . = To unsubscribe, E-mail to: archives: = http://mail.webdna.us/list/talk@webdna.us = Bug Reporting: = support@webdna.us >=20 > --------------------------------------------------------- This message = is sent to you because you are subscribed to the mailing list . To = unsubscribe, E-mail to: archives: = http://mail.webdna.us/list/talk@webdna.us = Bug Reporting: = support@webdna.us --Apple-Mail=_84F953D3-9AAF-4924-826B-6B34A4888859 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Some historical background, for those who=E2=80=99ve been = with the product since the WebCatalog days:

We used a bitwise ranking system = similar to this for keyword-based search results. WebCatalog=E2=80=99s = =E2=80=98native=E2=80=99 ranking (meaning the order in which it returned = search results if you *didn=E2=80=99t* specify a field to sort on) would = take all of your search keywords and assign them a bit, going from left = to right across a 32-bit unsigned integer. Each keyword that matched = would get a 1, and ones that didn=E2=80=99t match got a 0. The heuristic = assumption being that people would tend to type the most-important = keywords first, so those became the higher-order bits in the integer. = Each record in the table got a 32-bit score this way, and when we were = finished scanning the entire table, we merely sorted the results by that = 32-bit unsigned integer. Voila: best-matching records were always at the = top.

Not = exactly Google pagerank, but as heuristics go, it got the job done = fairly well ;)

On Sep = 30, 2016, at 10:47 AM, Patrick McCormick <duganmccormick@gmail.com> wrote:

It=E2=80=99s actually = called a bitwise operation - for geek history buffs. Let=E2=80=99s say = you have 5 different conditions you=E2=80=99re monitoring (I=E2=80=99m = just picking a random number of random tests):

1) temp above limit
2) ping fail
3)= http fail
4) dns fail
5) disk space = alarm

First, all of these must be yes/no, = binary conditions.  For temp limits, the set point and control loop = are elsewhere, this just looks at the resulting alarm as true/false.

If I put those conditions into a single field = called conditions, and none are in alarm, the result would simply be = five zeros. If any one condition alarms, the position of that number = would indicate what condition failed. if #1, 2, & 3 were all in = alarm, the result would 11100, or 7, in base 10.

You can either create a db to store all 32 possible different = combinations of alarms:

value= temppinghttpdnsspace
0fffff
1= t= f= f= f= f
2ftfff
etc.

You can write a function to trigger = whenever the [conditiona] variable is greater than zero, then lookup the = base 10 value of [conditions] in that db to see what the alarms are.

But because the data is = stored in binary, you can use a much quicker process to figure out which = conditions are in alarm. Take a look at the [listchars] command. You can do this:

[listchars = chars=3D[conditions]]
[index]: [char]
[/listchars]

That produces:

1: = 1
2: = 1
3: = 1
4: = 0
5: = 0

And, if you just store a = simple list of numbered alarms, the lookup happens in a fraction of the = time because your alarm db only has as many entries as alarms, rather = than all possible combinations.



On Sep 30, 2016, at 12:23 PM, dale = <dtherio@gmail.com> wrote:

Hello Patrick,

Interesting idea. = I've not tried that in the past. Any possible examples of how you would = do that? It definitely sounds like a good way to go.

Thank you,

Dale

On Fri, Sep 30, 2016 at 10:06 AM, Patrick = McCormick <duganmccormick@gmail.com> wrote:
I = think you should store each alarm condition as one bit in a binary = number long enough to accommodate all alarm conditions.  Then, each = base 10 value of that binary number represents a unique alarm or = combination of alarms.


> = On Sep 29, 2016, at 6:42 PM, dale <dtherio@gmail.com> wrote:
>
> I searched the talk archives, though I probably didn't = use the proper search terms.
>
> The = system for work I will be building is kind of like a fleet management = system. It isn't tracking the vehicles, but a computer and camera system = we install and maintain on the vehicles.
>
> One of the things I wanted to do is to put the various = problem codes (e.g. C HD PCB SSD) in a single field separated by a = comma. My thought is that it would be easier to search for systems = that had a specific problem and/or combination of problems.
>
> As it has been a few years since = coding in WebDNA, I'm not sure if this is the best way or not. We do = come up with new codes as time passes based on new systems.
>
> Would you recommend using a single = field for various codes as listed above where I would only have to = change the searches or using a hdr file to make adding additional codes = a bit easier?
>
> I really haven't = searched for multiple bits of data in a single field before.
>
> Ways in which this would be used:
> Someone checking the systems remotely would enter the = code for that system
> Techs fixing the system would = need to remove the appropriate codes for what they fixed
> Running reports to determine which systems have certain = issues
>
> Additionally, each time the = techs create a service ticket to make a repair, I will save a copy of = the the status of the system along with their notes in a history = database so we will have a life-cycle of the system. Those records = would only be editable by certain admin accounts.
>
> Thank you in advance for any recommendations as to the = best way to handle this.
>
> Dale
>
> = --------------------------------------------------------- This message = is sent to you because you are subscribed to the mailing list . To = unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us Bug = Reporting: support@webdna.us

---------------------------------------------------------
This message is sent to you because you are subscribed to
the mailing list <talk@webdna.us>.
To unsubscribe, E-mail = to: <talk-leave@webdna.us>
archives: http://mail.webdna.us/list/talk@webdna.us
Bug= Reporting: support@webdna.us

--------------------------------------------------------- = This message is sent to you because you are subscribed to the = mailing list . To unsubscribe, E-mail = to: archives: http://mail.webdna.us/list/talk@webdna.us Bug = Reporting: support@webdna.us

---------------------------------------= ------------------ This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us Bug Reporting: support@webdna.us

= --------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us Bug Reporting: support@webdna.us --Apple-Mail=_84F953D3-9AAF-4924-826B-6B34A4888859-- . Associated Messages, from the most recent to the oldest:

    
  1. Re: [WebDNA] Searching for multiple bits of data in one field or use multiple fields for each bit of data? ("C. Frank Wofle" 2016)
  2. Re: [WebDNA] Searching for multiple bits of data in one field or use multiple fields for each bit of data? (Grant Hulbert 2016)
  3. Re: [WebDNA] Searching for multiple bits of data in one field or use multiple fields for each bit of data? (christophe.billiottet@webdna.us 2016)
620 --Apple-Mail=_84F953D3-9AAF-4924-826B-6B34A4888859 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Some historical background, for those who=E2=80=99ve been with the = product since the WebCatalog days: We used a bitwise ranking system similar to this for keyword-based = search results. WebCatalog=E2=80=99s =E2=80=98native=E2=80=99 ranking = (meaning the order in which it returned search results if you *didn=E2=80=99= t* specify a field to sort on) would take all of your search keywords = and assign them a bit, going from left to right across a 32-bit unsigned = integer. Each keyword that matched would get a 1, and ones that didn=E2=80= =99t match got a 0. The heuristic assumption being that people would = tend to type the most-important keywords first, so those became the = higher-order bits in the integer. Each record in the table got a 32-bit = score this way, and when we were finished scanning the entire table, we = merely sorted the results by that 32-bit unsigned integer. Voila: = best-matching records were always at the top. Not exactly Google pagerank, but as heuristics go, it got the job done = fairly well ;) > On Sep 30, 2016, at 10:47 AM, Patrick McCormick = > wrote: >=20 > It=E2=80=99s actually called a bitwise operation - for geek history = buffs. Let=E2=80=99s say you have 5 different conditions you=E2=80=99re = monitoring (I=E2=80=99m just picking a random number of random tests): >=20 > 1) temp above limit > 2) ping fail > 3) http fail > 4) dns fail > 5) disk space alarm >=20 > First, all of these must be yes/no, binary conditions. For temp = limits, the set point and control loop are elsewhere, this just looks at = the resulting alarm as true/false. >=20 > If I put those conditions into a single field called conditions, and = none are in alarm, the result would simply be five zeros. If any one = condition alarms, the position of that number would indicate what = condition failed. if #1, 2, & 3 were all in alarm, the result would = 11100, or 7, in base 10. >=20 > You can either create a db to store all 32 possible different = combinations of alarms: >=20 > valuetemppinghttpdnsspace > 0fffff > 1tffff > 2ftfff > etc. >=20 > You can write a function to trigger whenever the [conditiona] variable = is greater than zero, then lookup the base 10 value of [conditions] in = that db to see what the alarms are. >=20 > But because the data is stored in binary, you can use a much quicker = process to figure out which conditions are in alarm. Take a look at the = [listchars] command. You can do = this: >=20 > [listchars chars=3D[conditions]] > [index]: [char] > [/listchars] >=20 > That produces: >=20 > 1: 1 > 2: 1 > 3: 1 > 4: 0 > 5: 0 >=20 > And, if you just store a simple list of numbered alarms, the lookup = happens in a fraction of the time because your alarm db only has as many = entries as alarms, rather than all possible combinations. >=20 >=20 >=20 > On Sep 30, 2016, at 12:23 PM, dale > wrote: >>=20 >> Hello Patrick, >>=20 >> Interesting idea. I've not tried that in the past. Any possible = examples of how you would do that? It definitely sounds like a good way = to go. >>=20 >> Thank you, >>=20 >> Dale >>=20 >> On Fri, Sep 30, 2016 at 10:06 AM, Patrick McCormick = > wrote: >> I think you should store each alarm condition as one bit in a binary = number long enough to accommodate all alarm conditions. Then, each base = 10 value of that binary number represents a unique alarm or combination = of alarms. >>=20 >>=20 >> > On Sep 29, 2016, at 6:42 PM, dale > wrote: >> > >> > I searched the talk archives, though I probably didn't use the = proper search terms. >> > >> > The system for work I will be building is kind of like a fleet = management system. It isn't tracking the vehicles, but a computer and = camera system we install and maintain on the vehicles. >> > >> > One of the things I wanted to do is to put the various problem = codes (e.g. C HD PCB SSD) in a single field separated by a comma. My = thought is that it would be easier to search for systems that had a = specific problem and/or combination of problems. >> > >> > As it has been a few years since coding in WebDNA, I'm not sure if = this is the best way or not. We do come up with new codes as time passes = based on new systems. >> > >> > Would you recommend using a single field for various codes as = listed above where I would only have to change the searches or using a = hdr file to make adding additional codes a bit easier? >> > >> > I really haven't searched for multiple bits of data in a single = field before. >> > >> > Ways in which this would be used: >> > Someone checking the systems remotely would enter the code for that = system >> > Techs fixing the system would need to remove the appropriate codes = for what they fixed >> > Running reports to determine which systems have certain issues >> > >> > Additionally, each time the techs create a service ticket to make a = repair, I will save a copy of the the status of the system along with = their notes in a history database so we will have a life-cycle of the = system. Those records would only be editable by certain admin accounts. >> > >> > Thank you in advance for any recommendations as to the best way to = handle this. >> > >> > Dale >> > >> > --------------------------------------------------------- This = message is sent to you because you are subscribed to the mailing list . = To unsubscribe, E-mail to: archives: = http://mail.webdna.us/list/talk@webdna.us = Bug Reporting: = support@webdna.us >>=20 >> --------------------------------------------------------- >> This message is sent to you because you are subscribed to >> the mailing list >. >> To unsubscribe, E-mail to: > >> archives: http://mail.webdna.us/list/talk@webdna.us = >> Bug Reporting: support@webdna.us >>=20 >> --------------------------------------------------------- This = message is sent to you because you are subscribed to the mailing list . = To unsubscribe, E-mail to: archives: = http://mail.webdna.us/list/talk@webdna.us = Bug Reporting: = support@webdna.us >=20 > --------------------------------------------------------- This message = is sent to you because you are subscribed to the mailing list . To = unsubscribe, E-mail to: archives: = http://mail.webdna.us/list/talk@webdna.us = Bug Reporting: = support@webdna.us --Apple-Mail=_84F953D3-9AAF-4924-826B-6B34A4888859 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Some historical background, for those who=E2=80=99ve been = with the product since the WebCatalog days:

We used a bitwise ranking system = similar to this for keyword-based search results. WebCatalog=E2=80=99s = =E2=80=98native=E2=80=99 ranking (meaning the order in which it returned = search results if you *didn=E2=80=99t* specify a field to sort on) would = take all of your search keywords and assign them a bit, going from left = to right across a 32-bit unsigned integer. Each keyword that matched = would get a 1, and ones that didn=E2=80=99t match got a 0. The heuristic = assumption being that people would tend to type the most-important = keywords first, so those became the higher-order bits in the integer. = Each record in the table got a 32-bit score this way, and when we were = finished scanning the entire table, we merely sorted the results by that = 32-bit unsigned integer. Voila: best-matching records were always at the = top.

Not = exactly Google pagerank, but as heuristics go, it got the job done = fairly well ;)

On Sep = 30, 2016, at 10:47 AM, Patrick McCormick <duganmccormick@gmail.com> wrote:

It=E2=80=99s actually = called a bitwise operation - for geek history buffs. Let=E2=80=99s say = you have 5 different conditions you=E2=80=99re monitoring (I=E2=80=99m = just picking a random number of random tests):

1) temp above limit
2) ping fail
3)= http fail
4) dns fail
5) disk space = alarm

First, all of these must be yes/no, = binary conditions.  For temp limits, the set point and control loop = are elsewhere, this just looks at the resulting alarm as true/false.

If I put those conditions into a single field = called conditions, and none are in alarm, the result would simply be = five zeros. If any one condition alarms, the position of that number = would indicate what condition failed. if #1, 2, & 3 were all in = alarm, the result would 11100, or 7, in base 10.

You can either create a db to store all 32 possible different = combinations of alarms:

value= temppinghttpdnsspace
0fffff
1= t= f= f= f= f
2ftfff
etc.

You can write a function to trigger = whenever the [conditiona] variable is greater than zero, then lookup the = base 10 value of [conditions] in that db to see what the alarms are.

But because the data is = stored in binary, you can use a much quicker process to figure out which = conditions are in alarm. Take a look at the [listchars] command. You can do this:

[listchars = chars=3D[conditions]]
[index]: [char]
[/listchars]

That produces:

1: = 1
2: = 1
3: = 1
4: = 0
5: = 0

And, if you just store a = simple list of numbered alarms, the lookup happens in a fraction of the = time because your alarm db only has as many entries as alarms, rather = than all possible combinations.



On Sep 30, 2016, at 12:23 PM, dale = <dtherio@gmail.com> wrote:

Hello Patrick,

Interesting idea. = I've not tried that in the past. Any possible examples of how you would = do that? It definitely sounds like a good way to go.

Thank you,

Dale

On Fri, Sep 30, 2016 at 10:06 AM, Patrick = McCormick <duganmccormick@gmail.com> wrote:
I = think you should store each alarm condition as one bit in a binary = number long enough to accommodate all alarm conditions.  Then, each = base 10 value of that binary number represents a unique alarm or = combination of alarms.


> = On Sep 29, 2016, at 6:42 PM, dale <dtherio@gmail.com> wrote:
>
> I searched the talk archives, though I probably didn't = use the proper search terms.
>
> The = system for work I will be building is kind of like a fleet management = system. It isn't tracking the vehicles, but a computer and camera system = we install and maintain on the vehicles.
>
> One of the things I wanted to do is to put the various = problem codes (e.g. C HD PCB SSD) in a single field separated by a = comma. My thought is that it would be easier to search for systems = that had a specific problem and/or combination of problems.
>
> As it has been a few years since = coding in WebDNA, I'm not sure if this is the best way or not. We do = come up with new codes as time passes based on new systems.
>
> Would you recommend using a single = field for various codes as listed above where I would only have to = change the searches or using a hdr file to make adding additional codes = a bit easier?
>
> I really haven't = searched for multiple bits of data in a single field before.
>
> Ways in which this would be used:
> Someone checking the systems remotely would enter the = code for that system
> Techs fixing the system would = need to remove the appropriate codes for what they fixed
> Running reports to determine which systems have certain = issues
>
> Additionally, each time the = techs create a service ticket to make a repair, I will save a copy of = the the status of the system along with their notes in a history = database so we will have a life-cycle of the system. Those records = would only be editable by certain admin accounts.
>
> Thank you in advance for any recommendations as to the = best way to handle this.
>
> Dale
>
> = --------------------------------------------------------- This message = is sent to you because you are subscribed to the mailing list . To = unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us Bug = Reporting: support@webdna.us

---------------------------------------------------------
This message is sent to you because you are subscribed to
the mailing list <talk@webdna.us>.
To unsubscribe, E-mail = to: <talk-leave@webdna.us>
archives: http://mail.webdna.us/list/talk@webdna.us
Bug= Reporting: support@webdna.us

--------------------------------------------------------- = This message is sent to you because you are subscribed to the = mailing list . To unsubscribe, E-mail = to: archives: http://mail.webdna.us/list/talk@webdna.us Bug = Reporting: support@webdna.us

---------------------------------------= ------------------ This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us Bug Reporting: support@webdna.us

= --------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us Bug Reporting: support@webdna.us --Apple-Mail=_84F953D3-9AAF-4924-826B-6B34A4888859-- . Grant Hulbert

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Authenticate (1997) help with writefile (1998) Donovans Theorem (2003) SSL PROBLEM (1999) WebCat2 - Getting to the browser's username/password data (1997) WCS Newbie question (1997) [AppendFile] problem (WebCat2b13 Mac .acgi) (1997) WebCat2 - [format thousands] (1997) Tech question about web dna actions (1998) RE: Adding headers to email (1997) Webcat 2.0.1b1 bug with IE 3.01/4.0p1 (1997) all records returned. (1997) sort without the or a (1998) emailer (1997) WebCatalog can't find database (1997) PCS Frames (1997) Unexpected error (1997) PCS Frames (1997) Request for page test (2003) Hiding HTML and page breaks (1997)