Pennock's Fiero Forum
  Announcements
  Totally new Search Engine!!!

Post New Topic  Post A Reply
Email This Page to Someone! | Printable Version


next newest topic | next oldest topic
Totally new Search Engine!!! by Cliff Pennock
Started on: 11-04-2001 06:25 PM
Replies: 35 (27 views)
Last post by: Cliff Pennock on 11-29-2001 06:36 PM
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-04-2001 06:25 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
After spending many nights of debugging, compiling, recompiling, some more debugging (well, you get the picture), I've implemented a complete new search engine for the forum.

The old search engine wasn't very good and was causing the server to almost grind to a halt when multiple searches were started. This was because there's over 2Gigabytes of data to search through and the old search was trying to do this by first copying everything to memory! Searching through the archives took forever, and some users mistook this for a non-responding search, so they started a new one which made the problem only worse.

The new search is fast, very fast. It is able to search through a section in less than a second. It has a few extra features over the old search, and a few other features have been dropped. BTW, did I mention the new search is fast?

I'm still testing and debugging, and also adding things, so you might see the search switch back and forth between the old and new one.

Enjoy!

[This message has been edited by Cliff Pennock (edited 11-04-2001).]

IP: Logged
PFF
System Bot
LZeitgeist
Member
Posts: 5662
From: Raleigh, NC, U.S.A.
Registered: Dec 2000


Feedback score:    (8)
Leave feedback





Total ratings: 126
Rate this member

Report this Post11-04-2001 06:57 PM Click Here to See the Profile for LZeitgeistSend a Private Message to LZeitgeistEdit/Delete MessageReply w/QuoteDirect Link to This Post
Sweet!!!!

Thank you, Cliff!!!!

------------------
Patrick W. Heinske -- LZeitgeist@aol.com
1988 Red Fiero Formula
- 1st Place - Stock Coupe - FOCOSEVA 2000
- 3rd Place - Stock Formula - FOCOA Nat'l 2001
- Class Winner - 1982 to Present - 2001 Tarheel Tigers All-Pontiac Show

IP: Logged
hugh
Member
Posts: 5563
From: Clementon,NJ,USA
Registered: Jun 2000


Feedback score: N/A
Leave feedback





Total ratings: 160
Rate this member

Report this Post11-04-2001 07:35 PM Click Here to See the Profile for hughSend a Private Message to hughEdit/Delete MessageReply w/QuoteDirect Link to This Post
Thanks!
IP: Logged
Raydar
Member
Posts: 40686
From: Carrollton GA. Out in the... country.
Registered: Oct 1999


Feedback score:    (13)
Leave feedback





Total ratings: 460
Rate this member

Report this Post11-04-2001 07:40 PM Click Here to See the Profile for RaydarSend a Private Message to RaydarEdit/Delete MessageReply w/QuoteDirect Link to This Post
WOW!

...and it's pretty fast, too!

------------------
Raydar - aka Steve

Some mornings, it just isn't worth chewing through the restraints.

IP: Logged
Nashco
Member
Posts: 4144
From: Portland, OR
Registered: Dec 2000


Feedback score:    (8)
Leave feedback





Total ratings: 74
Rate this member

Report this Post11-04-2001 08:29 PM Click Here to See the Profile for NashcoClick Here to visit Nashco's HomePageSend a Private Message to NashcoEdit/Delete MessageReply w/QuoteDirect Link to This Post
Hey, that's fast! Except...nothing popped up? I'm guessing it'll all smooth out soon enough. I'm very glad to see you've upgraded this! How hard would it be to add the ability to search through ALL sections, rather than one at a time? I have always wished we could do this, and since you're making changes.... Not complaining whatsoever, just asking what it would take, and if it's possible with your current rennovated version. You're the man Cliff

Bryce
88 GT

IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-04-2001 08:42 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
 
quote
Originally posted by Nashco:
Hey, that's fast! Except...nothing popped up?

That's weird. What are you searching for?

IP: Logged
Jefrysuko
Member
Posts: 3491
From: Oreana IL
Registered: Apr 2000


Feedback score:    (17)
Leave feedback





Total ratings: 108
Rate this member

Report this Post11-04-2001 09:00 PM Click Here to See the Profile for JefrysukoSend a Private Message to JefrysukoEdit/Delete MessageReply w/QuoteDirect Link to This Post
Wow that is fast. Do you have plans on having dates displayed also? If it comes down to dates or speed I will definetly take the speed but I liked knowing how recent something was with the previous search engine.
IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-04-2001 09:10 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
 
quote
Originally posted by Jefrysuko:
Do you have plans on having dates displayed also?

Yes, dates and originator will be included in the search results. It will not influence search speed in any way.

IP: Logged
Nashco
Member
Posts: 4144
From: Portland, OR
Registered: Dec 2000


Feedback score:    (8)
Leave feedback





Total ratings: 74
Rate this member

Report this Post11-04-2001 10:21 PM Click Here to See the Profile for NashcoClick Here to visit Nashco's HomePageSend a Private Message to NashcoEdit/Delete MessageReply w/QuoteDirect Link to This Post
 
quote
Originally posted by Cliff Pennock:
That's weird. What are you searching for?

Well, I found it's only symbols and numbers that don't work. I have been using the search a lot lately looking for info on 17" wheels. I typed in "17" and it came up with nothing, then " 17" " and nothing, then "17 wheels" and nothing, then just "wheels" and BAM, a lot. The reason it came up with nothing was given as:

"Sorry, nothing available for find=17
Search string may be too common or not a complete word."

Is there no way to search for numerical info, or symbols for that matter?

Bryce
88 GT

*typo edited

[This message has been edited by Nashco (edited 11-04-2001).]

IP: Logged
bHooper
Member
Posts: 4156
From: greensboro, nc
Registered: May 99


Feedback score: (1)
Leave feedback

Rate this member

Report this Post11-04-2001 10:21 PM Click Here to See the Profile for bHooperSend a Private Message to bHooperEdit/Delete MessageReply w/QuoteDirect Link to This Post
Good googilly moogilly! Thats fast!

Thanks, Cliff!

------------------
hoop
Red 86 GT 5 speed
Black 86 SE Automatic

IP: Logged
DJRice
Member
Posts: 2741
From: Merritt Island, FL USA
Registered: Jun 99


Feedback score: N/A
Leave feedback





Total ratings: 55
Rate this member

Report this Post11-04-2001 10:25 PM Click Here to See the Profile for DJRiceSend a Private Message to DJRiceEdit/Delete MessageReply w/QuoteDirect Link to This Post
I did a search for clutch and didnt get any hits....

Just Kidding! Man, Thats fast!

IP: Logged
PFF
System Bot
Neal
Member
Posts: 1247
From: Calgary AB, Canada
Registered: Jun 2001


Feedback score: N/A
Leave feedback

Rate this member

Report this Post11-05-2001 01:22 AM Click Here to See the Profile for NealClick Here to visit Neal's HomePageSend a Private Message to NealEdit/Delete MessageReply w/QuoteDirect Link to This Post
exelent job nice and fast

my request is for more than 10 hits per page, as many as 50 would be nice

IP: Logged
Cheever3000
Member
Posts: 12397
From: The Man from Tallahassee
Registered: Aug 2001


Feedback score: (1)
Leave feedback





Total ratings: 178
Rate this member

Report this Post11-05-2001 01:46 AM Click Here to See the Profile for Cheever3000Send a Private Message to Cheever3000Edit/Delete MessageReply w/QuoteDirect Link to This Post
I was much more successful with the old one. I used Subject Only vs. Entire Text a lot, as well as often knowing the username I was looking for, and even knowing how long ago. Based on my experiment a few moments ago, I'd vote on going back to the old one.
IP: Logged
Nashco
Member
Posts: 4144
From: Portland, OR
Registered: Dec 2000


Feedback score:    (8)
Leave feedback





Total ratings: 74
Rate this member

Report this Post11-05-2001 02:42 AM Click Here to See the Profile for NashcoClick Here to visit Nashco's HomePageSend a Private Message to NashcoEdit/Delete MessageReply w/QuoteDirect Link to This Post
THE OLD ONE??? With the increased search time, I can read through a few extra subjects as a sacrifice. I'd rather glance through an extra 50% more titles than wait a minute for the search. Maybe that's because I'm an internet freak though.. Slow connections drive me nuts!

I really did use the "search by username" function a lot though, that was a definite bonus to the old system.

Just me talking...I find it much more productive to search the entire text. Many threads change subject halfway through, or have a topic that doesn't fit the bill well.

Bryce
88 GT

[This message has been edited by Nashco (edited 11-05-2001).]

IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-05-2001 06:15 AM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
Ok, a few remarks.

Searching for "17" indeed did not yield any results because the system considered that too generic. In fact, the system considered every word with less than three characters as too generic. I've changed this so now you are able to search for stuff like "V8", "17" or "code 32" (where previously "32" would be discarded as a search term.

If you prefer the old search, then I'm afraid you are out of luck, since I will most certainly not go back to the old search. The old search was causing major problems and just can't cope with datasets as large as this forum's. In the past year, all problems of the forum being slow, or if you got some sort of "unreachable" error was because the old search was bringing the server to its knees. Like I said, the old search was trying to fit a 2Gb dataset into memory - not to mention it hijacked all processor usage.

You can still search for a specific user by just entering the username as a search term. The only reason the old search had a separate option for this, was because it used a different database when searching for posts made by a specific user to speed up the search. Since speed isn't an issue anymore, there is no separate option for it now.

The one option that's gone forever is searching for posts that are newer than a certain amount of time. The new search algorithms simply can't handle that.

Searching through all forums at once is an option I'm currently looking in to. The reason it's not implemented yet is because it requires an extra (very large) index.

And last, the search is still a work in progress so give it time to crystallize.

IP: Logged
TSPalmer
Member
Posts: 177
From: Franklinville, NC, USA
Registered: Mar 2001


Feedback score: N/A
Leave feedback

Rate this member

Report this Post11-05-2001 08:47 AM Click Here to See the Profile for TSPalmerSend a Private Message to TSPalmerEdit/Delete MessageReply w/QuoteDirect Link to This Post
Thanks Cliff!! The new search is 1000 times faster than the old.

------------------
T.Shawn Palmer
'85 SE 4cyl 5spd

IP: Logged
Nashco
Member
Posts: 4144
From: Portland, OR
Registered: Dec 2000


Feedback score:    (8)
Leave feedback





Total ratings: 74
Rate this member

Report this Post11-05-2001 12:53 PM Click Here to See the Profile for NashcoClick Here to visit Nashco's HomePageSend a Private Message to NashcoEdit/Delete MessageReply w/QuoteDirect Link to This Post
I dedicate a song from the movie Office Space by Ghetto Boys that I think describes the entire situation: "Damn It Feels Good To Be A Gangster"

Awesome Cliff...awesome!

Bryce
88 GT

IP: Logged
Archie
Member
Posts: 9436
From: Las Vegas, NV
Registered: Dec 1999


Feedback score:    (12)
Leave feedback





Total ratings: 546
Rate this member

Report this Post11-05-2001 05:22 PM Click Here to See the Profile for ArchieClick Here to visit Archie's HomePageSend a Private Message to ArchieEdit/Delete MessageReply w/QuoteDirect Link to This Post
 
quote
Originally posted by TSPalmer:
Thanks Cliff!! The new search is 1000 times faster than the old.

Boy that is fast..... my calculations show that it is 1001 time faster.

Good job.

Archie

IP: Logged
ray b
Member
Posts: 12441
From: miami
Registered: Jan 2001


Feedback score: N/A
Leave feedback





Total ratings: 325
Rate this member

Report this Post11-05-2001 06:16 PM Click Here to See the Profile for ray bSend a Private Message to ray bEdit/Delete MessageReply w/QuoteDirect Link to This Post
search button [icon] should it be more towards the center of the page and bigger???
in my normal window it is off the page on the far right unless i go to MAX SIZE or scroll over i can't see it and didnot know it was there untill i hunted for it.
maybe a better search needs a better location
and bigger size to be used more.
thanks for the improvements to a great site.

------------------
Question wonder and be wierd

IP: Logged
DRH
Member
Posts: 2683
From: Onalaska, WI, USA
Registered: Dec 1999


Feedback score: N/A
Leave feedback





Total ratings: 55
Rate this member

Report this Post11-05-2001 06:20 PM Click Here to See the Profile for DRHSend a Private Message to DRHEdit/Delete MessageReply w/QuoteDirect Link to This Post
The speed difference is incredible! I do seem to look for stuff I know I saw in a certain time frame though. If we can't search by date, any chance for an option to sort by date?


IP: Logged
theogre
Member
Posts: 32180
From: USA
Registered: Mar 99


Feedback score: N/A
Leave feedback





Total ratings: 567
Rate this member

Report this Post11-05-2001 07:11 PM Click Here to See the Profile for theogreClick Here to visit theogre's HomePageSend a Private Message to theogreEdit/Delete MessageReply w/QuoteDirect Link to This Post
Bloody hell that's fast... even has boolean functions..... Great Job!

As for the old system... Cliff's not kidding. You really don't want him going back to that thing. It was never meant to be used with something as big as this forum.

Study the help for the new system and you'll be able to find whatever you want quickly. If you want more practice with Boolean searching head over the Altavista. They support it in their advanced search as do most other good search systems.

Also, Use some common sense. Searching for something like a 2 digit number is kind of pointless. If you are looking for an error code, use code and the number as in Cliff's example. The more words you can think of relevant to your search, the better the results will come up. Another example... search for mirror and you'll be here till hell freezes looking at the results. Search for mirror turn signal in the archives and you get my mirror turn signal project on the first page....

------------------
11-Sept-01, The day the world as we knew it ended.

IP: Logged
PFF
System Bot
Monkeyman
Member
Posts: 15809
From: N. Wilkesboro, NC, USA
Registered: Nov 1999


Feedback score: (5)
Leave feedback





Total ratings: 182
Rate this member

Report this Post11-06-2001 04:57 AM Click Here to See the Profile for MonkeymanSend a Private Message to MonkeymanEdit/Delete MessageReply w/QuoteDirect Link to This Post
What's the score? I did a search for 'steering stabilizers' and came up with one thread. Under the thread title, it said something about 'score: 100'. The higher the score, the more free Fiero parts we're allowed?
IP: Logged
thomas_l
Member
Posts: 1197
From: Alpharetta, GA, US
Registered: Apr 2001


Feedback score: N/A
Leave feedback

Rate this member

Report this Post11-06-2001 04:38 PM Click Here to See the Profile for thomas_lClick Here to visit thomas_l's HomePageSend a Private Message to thomas_lEdit/Delete MessageReply w/QuoteDirect Link to This Post
Nice job! Very fast and sorted by relevance. Sorry to hear the date filter/sort is out. Tough to implement with a flat file database.
IP: Logged
DRH
Member
Posts: 2683
From: Onalaska, WI, USA
Registered: Dec 1999


Feedback score: N/A
Leave feedback





Total ratings: 55
Rate this member

Report this Post11-07-2001 09:00 AM Click Here to See the Profile for DRHSend a Private Message to DRHEdit/Delete MessageReply w/QuoteDirect Link to This Post
Cliff,

I noticed I can't search for things with a . (period), for example "4.9 swap", "2.8 turbo", etc...

Is there a fix or workaround?

Thanks, David

IP: Logged
Alex4mula
Member
Posts: 7403
From: Canton, MI US
Registered: Dec 1999


Feedback score:    (11)
Leave feedback





Total ratings: 153
Rate this member

Report this Post11-07-2001 12:38 PM Click Here to See the Profile for Alex4mulaSend a Private Message to Alex4mulaEdit/Delete MessageReply w/QuoteDirect Link to This Post
Could you implement the subject search? I used that most of the time to get threads that I know should talk specificaly about the tipic I wanted. Now if everybody has K&N or 3.4 in its signature then I can't do a search without getting almot all the database topics. Just my opinion.

------------------
Alex4mula
:) Red-Original Owner & White-1/94 with PM (:

IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-07-2001 12:40 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
I'm noting all suggestions and see what I can do.
IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-07-2001 05:38 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post

Cliff Pennock

11569 posts
Member since Jan 99
Added:
  • You can now search for stuff like "3.4" or "K&N". The search databases are currently being rebuild so give it about 15 minutes before you can actually find these things.
  • You know have a wildcard character. If you search for "steering*", it will find words like "steeringwheel", "steeringcolumn", etc.

I'm still looking at all the other requests.

[This message has been edited by Cliff Pennock (edited 11-07-2001).]

IP: Logged
JSocha
Member
Posts: 3522
From: Felton, MN, USA
Registered: Apr 2001


Feedback score: N/A
Leave feedback

Rate this member

Report this Post11-07-2001 05:59 PM Click Here to See the Profile for JSochaSend a Private Message to JSochaEdit/Delete MessageReply w/QuoteDirect Link to This Post
U-dah Mannnnnn!

------------------

IP: Logged
qwikgta
Member
Posts: 4659
From: Virginia Beach, VA
Registered: Jan 2001


Feedback score:    (21)
Leave feedback





Total ratings: 84
Rate this member

Report this Post11-07-2001 07:53 PM Click Here to See the Profile for qwikgtaSend a Private Message to qwikgtaEdit/Delete MessageReply w/QuoteDirect Link to This Post
I do like the new one, but I always used the "date" funcion so I was not looking at old posts, thanks cliff

Rob

IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-07-2001 07:54 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
Added:

  • You can now search in either the entire message, or the subject alone.
IP: Logged
ds21
Member
Posts: 301
From: Madison, WI
Registered: May 2001


Feedback score: N/A
Leave feedback

Rate this member

Report this Post11-07-2001 11:57 PM Click Here to See the Profile for ds21Send a Private Message to ds21Edit/Delete MessageReply w/QuoteDirect Link to This Post
getting close...comeon give it... give it...give us the date. Good work like it better allready!
IP: Logged
PFF
System Bot
Cheever3000
Member
Posts: 12397
From: The Man from Tallahassee
Registered: Aug 2001


Feedback score: (1)
Leave feedback





Total ratings: 178
Rate this member

Report this Post11-08-2001 12:08 AM Click Here to See the Profile for Cheever3000Send a Private Message to Cheever3000Edit/Delete MessageReply w/QuoteDirect Link to This Post
Thanks Cliff!!! I feel like I'm clutching my security blanket again.

IP: Logged
Alex4mula
Member
Posts: 7403
From: Canton, MI US
Registered: Dec 1999


Feedback score:    (11)
Leave feedback





Total ratings: 153
Rate this member

Report this Post11-08-2001 08:52 AM Click Here to See the Profile for Alex4mulaSend a Private Message to Alex4mulaEdit/Delete MessageReply w/QuoteDirect Link to This Post
Wow!! Thanks Cliff! That's fast customer service! Now I agree this is really better. Thanks again

------------------
Alex4mula
:) Red-Original Owner & White-1/94 with PM (:

IP: Logged
fierospeeder
Member
Posts: 1785
From: Illinois
Registered: Oct 1999


Feedback score: N/A
Leave feedback





Total ratings: 61
User Banned

Report this Post11-29-2001 01:56 AM Click Here to See the Profile for fierospeederClick Here to visit fierospeeder's HomePageSend a Private Message to fierospeederEdit/Delete MessageReply w/QuoteDirect Link to This Post
Cliff, i was wondering if we can search by username

and the items can be posted by date or revelance


plus, is there a time delay of when new posts are on the internet, and when the search is able to find those new posts.

it seems that i cant find something that was posted just recently

IP: Logged
Cheever3000
Member
Posts: 12397
From: The Man from Tallahassee
Registered: Aug 2001


Feedback score: (1)
Leave feedback





Total ratings: 178
Rate this member

Report this Post11-29-2001 05:42 PM Click Here to See the Profile for Cheever3000Send a Private Message to Cheever3000Edit/Delete MessageReply w/QuoteDirect Link to This Post
My searches last night didn't show anything more recent than August!!!
IP: Logged
Cliff Pennock
Administrator
Posts: 11569
From: Zandvoort, The Netherlands
Registered: Jan 99


Feedback score: (2)
Leave feedback





Total ratings: 698
Rate this member

Report this Post11-29-2001 06:36 PM Click Here to See the Profile for Cliff PennockClick Here to visit Cliff Pennock's HomePageSend a Private Message to Cliff PennockEdit/Delete MessageReply w/QuoteDirect Link to This Post
 
quote
Originally posted by Cheever3000:
My searches last night didn't show anything more recent than August!!!

That's because everything older than 3 months is moved to the Archives.

I will make some changes to the search engine once I have the time to do it. Hopefully, you will be able to look for posts by a specific user, and sort on date as well.

IP: Logged

next newest topic | next oldest topic

All times are ET (US)

Post New Topic  Post A Reply
Hop to:

Contact Us | Back To Main Page

Advertizing on PFF | Fiero Parts Vendors
PFF Merchandise | Fiero Gallery | Ogre's Cave
Real-Time Chat | Fiero Related Auctions on eBay



Copyright (c) 1999, C. Pennock