Hybrid Contextual Ontology-Fuzzy Logic and LSTM Model for Efficient Web Crawler Prediction and Traffic Optimization

The broad appeal of the World Wide Web (WWW) has presented challenges not only for society but also for the technological innovations that created and maintain it. The Web is expanding in society faster than we can comprehend its implications or establish guidelines for its use [1]. The widespread usage of the Internet has raised significant societal issues related to information availability, restrictions, and security. From a technical perspective, the Web's uniqueness and rapid growth have posed challenges in developing new applications that fully utilize its potential, as well as in building the infrastructure required to scale such applications to handle massive data loads and information sets [2].

Fields like data retrieval, computation, and decentralized networks have both benefited from and contributed to addressing these new challenges. These data repositories are now known as search engines, which index the entire WWW and present results to users based on their search queries [3]. Search engines employ web crawlers, also known as robots, to crawl the web and retrieve information from the WWW. These sources of knowledge include legal, social, educational, and food-related data [4].

A web crawler is an automated computer program that systematically searches the WWW. Web crawlers are also referred to by various names, including web robots, web spiders, web wanderers, bots, automated indexers, and spiders. Spidering is a common technique used by many websites, especially search engines, to gather current information [5]. Web crawlers are primarily employed to store copies of each web page they visit so that an internet search engine can later analyze and index the retrieved pages. Crawlers can also automate website maintenance tasks such as validating HTML code and checking links. They can be used to extract specific types of data, such as email addresses [6].

Generally, a web crawler starts with a list of URLs to visit, referred to as seed URLs. When the crawler visits these URLs, it identifies all the hyperlinks on the page and adds them to the crawl frontier—a list of URLs to visit in the future [7]. The crawler then continues to visit URLs from the frontier according to a set of policies. The effectiveness of the WWW can be attributed to two key factors: its vastness and the lack of centralized content management. These same factors also contribute to the challenges of finding information on the Web [8]. The Web presents a challenge to traditional Information Retrieval (IR) techniques because the sheer volume and rapid evolution of the Web exceed the capacity of modern search engines [9]. The quality of content on the Web is uneven, with a small number of valuable web pages buried under a massive amount of less relevant content.

For the Internet to function correctly, certain protocols must be followed. A protocol is a well-defined format for connecting, communicating, and exchanging data between two machines on a network [10]. The Internet employs several important protocols, including Hypertext Transfer Protocol (HTTP), Hypertext Transfer Protocol Secure (HTTPS), and File Transfer Protocol (FTP). HTTPS is used by websites that request sensitive information such as passwords and other private data [11]. HTTP is used to transfer web pages over the WWW. Websites using HTTPS have a Secure Sockets Layer (SSL) certificate issued by a trusted third party, which can be examined when a webpage is accessed using HTTPS. FTP is another common protocol used for transferring files across the Internet [12]. Web developers use FTP to upload new versions of their websites, while HTTP is used to display files in a web browser. FTP allows files to be transferred from one machine to another, such as from a home computer to a remote web server [13]. Every search engine maintains a core database of websites. The search engine retrieves results from this database based on the user's query. A web crawler, also known as a spider or bot, searches the Internet for web content. The web crawler begins by downloading online documents from an initial set of URLs [14].

1.png

Figure 1. Web crawler architecture

The search engine's information is kept up to date by the web crawler architecture integrates updates from the internet. To overcome computational congestion, multiple instances of this component operate in parallel and communicate at high frequencies. Using the HTTP protocol, web crawlers retrieve and manipulate web pages from the internet [15]. The web crawler architecture consists of two main components: the processor and the downloader. The downloader retrieves websites from the World Wide Web and sends them to the processor for further analysis. The HTTP protocol is used to download these web pages. Various efficiency techniques such as TCP reuse, pipelining, conditional GETs, variable refresh rates, and DNS caching are employed to overcome network constraints [16]. Simultaneous web crawlers operate concurrently to optimize download speed and cover a large portion of the internet, as shown in Figure 1. Concurrent crawlers face challenges such as webpage overlap, bandwidth consumption, and reliability issues [17]. Since crawlers might not be aware of each other's previous searches, the same web pages may be retrieved multiple times, resulting in overlap. Minimizing this overlap conserves network bandwidth, allows for more unique downloads, and improves the overall efficiency of web crawlers [18].

The goal of a web crawler is to download as many significant web pages as possible. When multiple crawlers operate simultaneously, they may lack knowledge of the complete set of pages retrieved collectively. Consequently, crawling decisions based solely on local data can lead to poor performance [19]. To reduce redundancy and maintain the quality of retrieved content, crawlers must communicate with each other. However, this communication consumes both bandwidth and time.

Crawlers assign priority to web pages based on their relevance score, which is calculated using techniques such as Term Frequency–Inverse Document Frequency (TF-IDF) and cosine similarity [20].

Focuses on enhancing the efficiency and accuracy of web crawling in dynamic and complex web environments. Existing web crawlers often struggle with predicting changes in web content and managing network traffic, which can lead to inefficiencies in data retrieval and indexing. This model combines the strengths of contextual ontology, fuzzy logic and LSTM networks to address these challenges shown in Figure 2. The contextual ontology component is designed to capture and organize the relevant context from web pages, improving the system’s ability to retrieve meaningful and precise information. Fuzzy logic is integrated to manage uncertainty and imprecision in the web data, enabling more adaptive and efficient decision-making in web crawling processes. LSTM networks are utilized to predict web crawler behavior by learning from past interactions allowing the system to anticipate changes in web content and optimize its crawling strategies. This hybrid proposed system aims to reduce unnecessary network traffic, enhance prediction accuracy, and improve the overall performance of web crawlers. By addressing key issues such as traffic optimization and crawler prediction, this model offers a more intelligent and scalable solution for web indexing and information retrieval in the fast-evolving digital landscape.

2.png

Figure 2. Proposed architecture

3.1 Dataset description

Large-scale web page material such as HTML, text, and metadata included in the Web Crawl Data 1 collection was obtained through Common Crawl. The algorithm's capacity to efficiently gather and evaluate material from a variety of web pages is trained using this dataset. The history of web page modifications may be found in the Web Page Change Logs dataset, which was retrieved from Archive.org. The information in the dataset is utilized for training the LSTM network allowing the algorithm to forecast upcoming shifts in online content according to historical trends. Variables such as URLs, time stamps, and modification frequency are included in the dataset. Comprehensive information on network traffic such as bandwidth utilization, number of requests and frequency latency available in the Network Traffic Logs dataset was gathered using custom crawlers. Using fuzzy reasoning to maximize network traffic and cut down on wasteful bandwidth usage requires this set of data.

Contextual connections and semantic linkages among the organized information provided by the Ontology-Based Dataset found in resources such as DBpedia and Wikipedia. Through a contextual and ontological framework, this collection of data offers a comprehensive contextual foundation that enhances web indexing. Finally, contextual information such as phrases and context labels are included in the Web Page Context Information collected by specialized crawlers. This collection of data uses context-based insight to better analyze and index online material helps to improve information retrieval shown in Table 1. These datasets aid in the learning, optimization, and validation of many facets of the approach, guaranteeing thorough development and efficient operation in the areas of handling traffic and web crawler predictions.

The information that is contained throughout each dataset is exemplified by these examples include web page pleased, change history, structured knowledge, network traffic statistics derived from ontologies, and contextual data taken from websites shown in Tables 2-6.

Table 1. Dataset description

Table 2. Web crawl data 1

Table 3. Web page change logs

Table 4. Network traffic logs

Table 5. Ontology-based dataset

Table 6. Web page context data

3.2 Pre-processing

For the development of a contextual ontology-based fuzzy logic and LSTM model for web crawler prediction, data preprocessing involves several steps to clean and prepare web data.

3.2.1 HTML tag removal

Extract raw text from HTML content to focus on the actual textual information described using a regular expression (regex):

Regex Pattern: <[^>]+>

This pattern matches any sequence of characters that starts < with and ends with >, effectively identifying and removing HTML tags.

Example: Original HTML: <html><head><title>Ex</title></head><body><p>Sample text.</p></body></html>

Cleaned Text: Sample text.

HTML Tag Removal: Utilize regex <[^>]+> to strip HTML tags from content.

3.2.2 Regular expression pattern matching

Extract specific patterns or clean up the text further based on defined patterns.

The regex pattern itself is used for text processing.

Example Patterns: Email Extraction: [\w\.-]+@[\w\.-]+

Matches email addresses. URL Extraction: http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+

Matches URLS.

Example:

Text: Contact us at example@ex.com or visit http://ex.com

Extracted Information: ex@ex.com, http://ex.com

Regular Expression Pattern Matching: Apply regex patterns to extract or clean specific information (e.g., emails, URLs).

3.2.3 Stop word removal

Remove common words (stop words) that do not contribute significant meaning to the text, thereby reducing dimensionality and noise.

Stop Words List Example: ['a', 'an', 'the', 'is', 'in', 'and']

Process: Tokenize the text into words. Remove words that are in the stop words list.

Example:

Original Text: The quick brown fox jumps over the lazy dog.

Tokenized: [The', 'quick', 'brown', 'fox', 'Jumps', 'over', 'the', 'lazy', 'dog']

After Stop Word Removal: ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog]

Stop Word Removal: Filter out common words from the tokenized text using a predefined list of stop words.

These preprocessing steps ensure that the data fed into the contextual ontology-based fuzzy logic and LSTM model is clean, relevant, and ready for further analysis and modeling.

3.3 Feature extraction

The goal is to maximize the probability of observed word-context pairs while minimizing the probability of non-observed pairs. This can be mathematically represented in the objective function:

$\begin{gathered} { Objective\ Function }=  \sum_{(w, c) \in D} \log p(Z=1 \mid(w, c))+  \sum_{(w, c) \in D^{\prime}} \log p(Z=0 \mid(w, c))\end{gathered}$          (1)

To model these probabilities, we often use a sigmoid function for $p(Z=1 \mid(w, c))$:

$p(Z=1 \mid(w, c))=\sigma\left(v_w^T v_c\right)$          (2)

where, $\sigma$ denotes the sigmoid function:

$\sigma(i)=\frac{1}{1+e^{-i}}$          (3)

Similarly, $p(Z=1 \mid(w, c))$ can be computed as:

$p(Z=1 \mid(w, c))=1-\sigma\left(v_w^T v_c\right)$          (4)

The optimization process involves adjusting the vector representations v_w and v_c to maximize the objective function, thereby learning meaningful word embeddings based on their contexts. Figure 3 explains the preprocessing and feature extraction in step by step procedure. For the Development of a contextual ontology-based fuzzy logic and LSTM Model for web crawler prediction, feature extraction using LSTM networks involves transforming raw data into meaningful features that capture temporal dependencies and contextual information. 

3.png

Figure 3. Preprocessing and feature extraction

Step 1: Data Preparation: Transform raw web data into a sequence suitable for LSTM input.

Steps:

Tokenization: Convert raw text into sequences of tokens (words or subwords).

Embedding: Use word embeddings (e.g., Word2vec, Glove) to represent tokens as dense vectors.

$E(w)=v_w$          (5)

where, E(w) is the embedding of word w. v_w is the pre-trained vector representation of w.

Step 2: LSTM Feature Extraction: LSTM networks are designed to capture temporal dependencies in sequential data. The LSTM unit includes several components to process the sequence data:

Forget Gate: $f_t=\sigma\left(W_f .\left[h_{t-1}, i_t\right]+b_f\right)$          (6)

Input Gate: $x_t=\sigma\left(W_x .\left[h_{t-1}, i_t\right]+b_x\right)$          (7)

$\bar{C}_t=\tanh \left(W_C \cdot\left[h_{t-1}, i_t\right]+b_C\right)$          (8)

Cell State Update: $C_t=f_t \cdot C_{t-1}+x_t \cdot \bar{C}_t$          (9)

Output Gate: $o_t=\sigma\left(W_o .\left[h_{t-1}, i_t\right]+b_o\right)$          (10)

$h_t=o_t \cdot \tanh \left(C_t\right)$          (11)

where, $\sigma$ is the sigmoid activation function; tanh is the hyperbolic tangent activation function; $W_f, W_C, W_x, W_o$ are weight matrices for the forget, input, cell, and output gates, respectively; $b_f, b_C, b_x, b_o$ are bias terms; $i_t$ is the input vector at time $\mathrm{t} ; h_{t-1}$ is the hidden state from the previous time step; $C_t$ is the cell state at time f.

Step 3: Contextual Features Extraction: Integrate contextual information extracted from a contextual ontology into the LSTM features.

Contextual Embedding: Contextual embeddings represent words in the context of their surrounding words, derived from a contextual ontology.

$E_c(w)=v_w+c_w$          (12)

where, $E_c(w)$ is the contextual embedding of word w;  is the vector representing contextual information from the ontology.

Feature Vector for LSTM: Combine contextual embeddings with LSTM outputs to form the final feature vector.

$F_t={Concat}\left(h_t, E_c\left(w_t\right)\right)$          (13)

where, $F_t$ is the feature vector at time t. Concat denotes concatenation of LSTM hidden state  and contextual embedding $E_c\left(w_t\right)$.

Step 4: Feature Aggregation: Aggregate features from multiple time steps to represent the entire sequence.

Pooling: Apply pooling techniques (e.g., max pooling, average pooling) to aggregate features over time.

$F_{ {agg }}={Pooling}\left(\left\{F_t\right\}_{t=1}^T\right)$          (14)

where, $F_{ {agg }}$ is the aggregated feature vector. $\left\{F_t\right\}_{t=1}^T$ represents the feature vectors from all time steps. 

Final Feature Representation: The aggregated features can be used for further processing, such as classification or prediction.

Feature extraction involves preparing data through tokenization and embedding, processing sequences with LSTM to capture temporal dependencies, integrating contextual information from ontology-based embeddings, and aggregating features for final representation. The resulting features are then utilized for web crawler prediction and other analyses shown in Figure 4.

Input Data: This is the input data for the model can be text, images, or other sequential data.

Pre-processing: This step involves cleaning and preparing the input data. For text data, this might include tasks like tokenization, stemming, and stop word removal.

Word Embedding: This layer converts words or tokens into numerical representations, capturing semantic relationships between words.

4.png

Figure 4. Feature extraction using LSTM

Convolution and Max-pooling: These layers are typically used in CNNs for extracting features from the input data. Convolutional layers apply filters to the input data, while max-pooling layers downsample the output to reduce computational cost and preserve important features.

LSTM Layers: These layers are recurrent neural networks that are particularly effective at capturing long-term dependencies in sequential data. They use a gating mechanism to control the flow of information through the network, allowing them to remember information over long periods of time.

Fully Connected Layer: This layer combines the outputs from the LSTM layers into a fixed-length vector.

Softmax Layer: This layer applies the softmax function to the output of the fully connected layer, producing a probability distribution over the possible classes.

Classification: The final output of the model is the predicted class based on the probability distribution produced by the softmax layer.

Figure 4 explains the architecture is well-suited for tasks that involve classifying sequential data, such as text classification or time series analysis. CNN with LSTMs allows the model to capture both local and global features in the input data, making it a powerful tool for a variety of applications.

3.4 Fuzzy ontology-based semantic model for a web crawler

To simplify the realization of more fuzzy functions, the proposed architecture shown in Figure 5 includes preparing and after-processing elements in addition to fuzzy representations and identification elements. The fundamental supporting module for fuzzy operations, comprising fuzzy logic, regulations, and computations called the fuzzy engine. Service Fuzzalizer fuzzals up descriptions of internet services into fuzzy forms. The Fuzzy Ontology Knowledge Base (FOKB) where fuzzy service ontologies are kept. This database can be kept alone or integrated into the UDDI register center with a UDDI extension. Request Fuzzalizer filters user requests before converting them into fuzzy forms for matching. Fuzzy Matchmaking is used to carry out service finding. Requesters' usage data is recorded in Usage Record and can be utilized for user-chosen matching. The first step in utilizing the framework for service discovery is to translate the service representation into more ambiguous language. Provision of services Web services are transformed into fuzzy ontologies by Fuzzalizer and stored in FOKB. Preprocessing like this can be done after the service is advertised to prevent sluggish runtime performance. The framework uses the industry-standard UDDI to find online services.

Users can use clear or vague language when requesting services. The requests Fuzzalizer converts crisp phrases used in a service consumer's query to fuzzy forms upon inquiry initiation. Additionally, the person making the request Fuzzalizer standardizes fuzzy requests. Fuzzy matching performs fuzzy service matching using items described in fuzzy ontologies. The fuzzy concepts are related to one another via approximation reasoning. The user's Account and Usage History will get the match outcome for service activation and customer preference records, respectively. To expedite finding services, the choice records will be utilized as the default information for subsequent requests. Fuzzy thresholds or matching weights can be changed and a fresh match is made if the consumer is not happy with the matching service. Using the above framework, customers may quickly modify their selection of services based on personal choice and request services using ambiguous language. By combining fuzzy logic with ontology, Fuzzy Ontology Rules with Equations address ambiguity and uncertainty in categorizing information or decision-making procedures. These rules leverage both fuzzy logic and semantic connections (ontology) to enhance the classification of information in the surroundings of web crawlers and other operations. The Fuzzy Ontology-Based Semantic Model for Web Crawler combines fuzzy logic and ontology to handle uncertain and imprecise data in web crawling. The ontology defines the relationships between concepts on the web, while fuzzy logic manages ambiguity in the web data.

5.png

Figure 5. Fuzzy ontology framework for web service discovery

3.4.1 Ontology structure

An ontology is structured as a set of concepts C, relationships R, and instances I.

Concepts: The primary topics or categories.

Relationships: Connections between different concepts.

Ontology is represented as:

$O=(C, R, I)$          (15)

where, $C=\left\{c_1, c_2, \ldots, c_n\right\}$ is a set of concepts. $R=$ $\left\{r_1, r_2, \ldots, r_m\right\}$ is a set of relationships. $I=\left\{i_1, i_2, \ldots, i_p\right\}$ is a set of instances.

In fuzzy logic form, this can be represented as:

$\begin{gathered}\mu_{ {medium\ relevance }}(p)=  \min \left(\mu_{M L_{{category }}}(p), \mu_{ {content\ relevance }}(p)\right)\end{gathered}$          (16)

where, $\mu_{M L_{ {category }}}(p)$ represents the fuzzy membership degree of the web page in the "Machine Learning" category within the ontology. $\mu_{ {content\ relevance }}(p)$ is the fuzzy membership value of the page content relevance.

3.4.2 Combining fuzzy rules using fuzzy inference

The output from multiple fuzzy rules is combined using fuzzy aggregation methods, such as min and max operators. For example, consider the two rules from above:

Rule 1 provides a high relevance score with a degree $\mu_{ {high }}(p)$.

Rule 2 provides a medium relevance score with a degree $\mu_{ {medium }}(p)$.

The combined relevance score $\mu_{ {combined }}(p)$ can be computed using:

$\mu_{ {combined }}(p)=\max \left(\mu_{\text {high }}(p), \mu_{ {medium }}(p)\right)$          (17)

This takes the maximum of the two membership values, assigning the higher relevance to the page.

3.4.3 Defuzzification

Once the fuzzy inference is complete, the result is defuzzified to obtain a crisp relevance score that can be used for ranking or decision-making. For example, if the output relevance score $\mu_{ {combined }}(p)$ falls within the fuzzy sets $R_{ {high }}$, $R_{ {medium }}$ and $R_{ {low }}$, the final relevance score can be obtained using centroid or mean of maxima methods.

$R_{ {final }}(p)=\frac{\sum_{x=1}^n \mu_x(p) . r_x}{\sum_{x=1}^n \mu_x(p)}$          (18)

where, $\mu_x(p)$ is the membership degree of page p in fuzzy set x. $r_x$ is the representative crisp value of set x (e.g., 1 for high, 0.5 for medium, 0 for low).

3.4.4 Semantic matching with fuzzy ontology

The semantic relevance of a web page p to a query q is evaluated using fuzzy ontology-based semantic matching. The process involves calculating the degree to which the web page's content matches the ontological concepts related to the query.

$R(q, p)=\frac{\sum_{x=1}^n \mu_p\left(c_x\right) \cdot \mu_q\left(c_x\right)}{\sum_{x=1}^n \mu_q\left(c_x\right)}$          (19)

where: $R(q, p)$ is the relevance score of the web page to the query q. $\mu_p\left(c_x\right)$ is the membership degree of concept  for web page p. $\mu_q\left(c_x\right)$ is the membership degree of concept  for query q.

3.4.5 Fuzzy Inference System (FIS)

To model the semantic relevance of web pages, a fuzzy inference system can be implemented. This FIS takes the inputs (semantic similarity, page content features, etc.) and processes them through fuzzy rules to determine the relevance score.

3.4.6 Aggregating results in web crawling

The final step is aggregating the relevance scores across different web pages and returning the most relevant results.

$R_{ {total }}=\sum_{k=1}^m w_k \cdot R\left(q \cdot p_k\right)$          (20)

where, $R_{ {total }}$ is the total relevance score. $w_k$ is the weight assigned to each page $p_k$, which can be determined based on factors such as authority or content quality These equations represent the core mechanisms of the Fuzzy Ontology-Based Semantic Model and demonstrate how fuzzy logic is used to evaluate the relevance of web content in a way that accounts for ambiguity and semantic relationships.

The proposed incremental parallel web crawler shown in Figure 6. The architecture's primary coordinating element is the Multi Threaded (MT) server. It maintains a connection pool with computer clients that download the web pages instead of downloading any paperwork on its own. The term client crawlers refers to all the many ways that clients communicate with one another via servers. Depending on the extent of the real implementation and the available resources, the number of customers may change. It should be mentioned that all communication between consumer crawlers occurs through the server because there are no direct linkages between them. Only the modified papers are thereafter saved in the repositories in a searchable or insertable format after the modification detecting module assists in determining if the intended page has changed or not. As a result, the archive is updated with the most recent data that is accessible at the search engine databases endpoint.

6.png

Figure 6. Architecture of incremental parallel web crawler

Algorithm: Fuzzy ontology-based semantic model with LSTM

Input: $W=\left\{w_1, w_2, \ldots, w_n\right\}$: Set of webpages; $Q=\left\{q_1, q_2, \ldots, q_n\right\}$: Query terms; $T_c$: Training corpus (webpage content and context data); O: Ontology (set of domain knowledge and semantic relationships); F: Fuzzy rules; E: Embedding matrix for LSTM input; LSTM: Long Short-Term Memory network for time-series prediction.

Output: Predicted relevance and ranking score for each webpage $w_x$; Traffic optimization and next crawling step.

Step 1: Preprocess Webpage Data

Step 1.1: Remove HTML tags using regular expressions:

 $clean_-content\left(w_x\right) regex.sub \left(<, * ?>, ", w_x\right)$

where, regex.sub removes the HTML tags.

Step 1.2: Remove stopwords from each webpage content:

$w_x^{{clean }}=w_x^{{cotent }} \backslash stopwords$          (21)

Step 2: Contextual Ontology Analysis: For each webpage $w_s$:

Extract keywords K ($w_s$) from the webpage content.

Map extracted keywords to concepts in the ontology 0:

$C\left(w_x\right) {map}\left(K\left(w_x\right), 0\right)$          (22)

Calculate semantic similarity between webpage keywords and query:

$\mu_{ {semantic }}\left(w_x\right)={cosine}\left(K\left(w_x\right), Q\right)$

where, cosine similarity measures the semantic relationship between webpage content and the query.

Step 3: Apply Fuzzy Logic Rules

Step 3.1: Define fuzzy membership functions for relevance

$\mu_R\left(w_x\right)=\frac{1}{1+e^{-k\left(i-i_o\right)}}$          (24)

where, k and $i_o$ parameters defining the fuzzy function's steepness and midpoint, respectively.

Step 3.2: Apply fuzzy rules:

IF $\mu_{{semantic }}\left(w_x\right)$ is high AND $C\left(w_x\right)$ matches the ontology:

$\begin{aligned} & \mu_{ {relevance }}\left(w_x\right)  =\min \left(\mu_{ {semantic }}\left(w_x\right), \mu_{ {ontology }}\left(C\left(w_x\right)\right)\right)\end{aligned}$          (25)

Combine fuzzy rules for all conditions using fuzzy inference.

Step 4: Feature Representation with LSTM

Step 4.1 Embed webpage content into vector space using word embeddings (e.g., Word2Vec or Glove):

$E\left(w_x\right)= Embed \left(w_x^{\text {clean }}\right)$          (26)

Step 4.2 Feed embedded vectors into the LSTM network to model time-dependent crawling behavior:

Step 4.3 Initialize LSTM weights.

For each webpage sequence $\left\{w_1, w_2, \ldots, w_t\right\}$:

$h_t=\sigma\left(W_h h_{t-1}+W_\tau E\left(w_t\right)+b\right)$          (27)

where, $h_t$ is the hidden state, $W_h, W_i$are weight matrices, and b is the bias term.

Step 4.4 Output layer of the LSTM provides the prediction score:

$j_t=={softmax}\left(W_j h_t+b_j\right)$          (28)

where,  is the relevance score for each webpage at time t.

Step 5: Combine Fuzzy Logic and LSTM Predictions: For each webpage $W_t$, the final prediction score is a weighted combination of fuzzy relevance and LSTM output:

${Score}\left(w_x\right)=\alpha \cdot \mu_{ {relevance }}\left(w_x\right)+(1-\alpha) \cdot j_x$          (29)

where, $\alpha$ is a weight balancing the fuzzy logic and LSTM outputs.

Step 6: Web Crawler Traffic Optimization: Optimize the crawling process by ranking webpages based on the final score Score($w_x$). Prioritize the next crawl based on the predicted relevance and system traffic load:

IF Score($w_x$) is high AND system traffic is low:

${Crawl}\left(w_x\right)$ next

ELSE delay the crawl.

Step 7: Repeat and Update: Update the ontology and fuzzy logic rules based on new crawled data and user feedback. Retrain the LSTM model periodically using updated data.

Equations showing the relationships between contextual ontology, fuzzy logic, and LSTM are used in this approach to create a sophisticated web crawler prediction system. The crawler's operations are guided by the ultimate relevancy score maximizes network traffic and relevancy.